Question: Represent the text to find a very short summary (1-2 sentences max)

Question: A major goal of unsupervised learning is to discover data representations that are useful for subsequent tasks, without access to supervised labels during training. Typically, this involves minimizing a surrogate objective, such as the negative log likelihood of a generative model, with the hope that representations useful for subsequent tasks will arise as a side effect. In this work, we propose instead to directly target later desired tasks by meta-learning an unsupervised learning rule which leads to representations useful for those tasks. Specifically, we target semi-supervised classification performance, and we meta-learn an algorithm -- an unsupervised weight update rule -- that produces representations useful for this task. Additionally, we constrain our unsupervised update rule to a be a biologically-motivated, neuron-local function, which enables it to generalize to different neural network architectures, datasets, and data modalities. We show that the meta-learned update rule produces useful features and sometimes outperforms existing unsupervised learning techniques. We further show that the meta-learned unsupervised update rule generalizes to train networks with different widths, depths, and nonlinearities. It also generalizes to train on data with randomly permuted input dimensions and even generalizes from image datasets to a text task. Supervised learning has proven extremely effective for many problems where large amounts of labeled training data are available. There is a common hope that unsupervised learning will prove similarly powerful in situations where labels are expensive, impractical to collect, or where the prediction target is unknown during training. Unsupervised learning however has yet to fulfill this promise. One explanation for this failure is that unsupervised representation learning algorithms are typically mismatched to the target task. Ideally, learned representations should linearly expose high level attributes of data (e.g. object identity) and perform well in semi-supervised settings. Many current unsupervised objectives, however, optimize for objectives such as log-likelihood of a generative model or reconstruction error, producing useful representations only as a side effect. Unsupervised representation learning seems uniquely suited for meta-learning BID0 ). Unlike most tasks where meta-learning is applied, unsupervised learning does not define an explicit objective, which makes it impossible to phrase the task as a standard optimization problem. It is possible, however, to directly express a meta-objective that captures the quality of representations produced by an unsupervised update rule by evaluating the usefulness of the representation for candidate tasks. In this work, we propose to meta-learn an unsupervised update rule by meta-training on a meta-objective that directly optimizes the utility of the unsupervised representation. Unlike hand-designed unsupervised learning rules, this meta-objective directly targets the usefulness of a representation generated from unlabeled data for later supervised tasks. By recasting unsupervised representation learning as meta-learning, we treat the creation of the unsupervised update rule as a transfer learning problem. Instead of learning transferable features, we learn a transferable learning rule which does not require access to labels and generalizes across both data domains and neural network architectures. Although we focus on the meta-objective of semi-supervised classification here, in principle a learning rule could be optimized to generate representations for any subsequent task. Unsupervised learning is a topic of broad and diverse interest. Here we briefly review several techniques that can lead to a useful latent representation of a dataset. In contrast to our work, each method imposes a manually defined training algorithm or loss function whereas we learn the algorithm that creates useful representations as determined by a meta-objective. Autoencoders work by first compressing and optimizing reconstruction loss. Extensions have been made to de-noise data (; 2010), as well as compress information in an information theoretic way . further explored scaling up these unsupervised methods to large image datasets. Generative adversarial networks take another approach to unsupervised feature learning. Instead of a loss function, an explicit min-max optimization is defined to learn a generative model of a data distribution. Recent work has shown that this training procedure can learn unsupervised features useful for few shot learning (; ;).Other techniques rely on self-supervision where labels are easily generated to create a non-trivial'supervised' loss. Domain knowledge of the input is often necessary to define these losses. use unscrambling jigsaw-like crops of an image. Techniques used by and rely on using temporal ordering from videos. Another approach to unsupervised learning relies on feature space design such as clustering. showed that k-means can be used for feature learning. jointly learn features and cluster assignments. develop a scalable technique to cluster by predicting noise. Other techniques such as , , and define various desirable properties about the latent representation of the input, such as predictability, complexity of encoding mapping, independence, or sparsity, and optimize to achieve these properties. Most meta-learning algorithms consist of two levels of learning, or'loops' of computation: an inner loop, where some form of learning occurs (e.g. an optimization process), and an outer loop or metatraining loop, which optimizes some aspect of the inner loop, parameterized by meta-parameters. The performance of the inner loop computation for a given set of meta-parameters is quantified by a meta-objective. Meta-training is then the process of adjusting the meta-parameters so that the inner loop performs well on this meta-objective. Meta-learning approaches differ by the computation performed in the inner loop, the domain, the choice of meta-parameters, and the method of optimizing the outer loop. Some of the earliest work in meta-learning includes work by , which explores a variety of meta-learning and self-referential algorithms. Similarly to our algorithm, Bengio et al. (1990; 1992) propose to learn a neuron local learning rule, though their approach differs in task and problem formulation. meta-learn supervised learning rules which mix local and global network information. A number of papers propose meta-learning for few shot learning (; ; ; ;), though these do not take advantage of unlabeled data. Others make use of both labeled and unlabeld data . uses a task created with no supervision to then train few-shot detectors. use meta-learning for unsupervised learning, primarily in the context of clustering and with a small number of meta-parameters. Figure 1: Left: Schematic for meta-learning an unsupervised learning algorithm. The inner loop computation consists of iteratively applying the UnsupervisedUpdate to a base model. During metatraining the UnsupervisedUpdate (parameterized by θ) is itself updated by gradient descent on the MetaObjective. Right: Schematic of the base model and UnsupervisedUpdate. Unlabeled input data, x 0, is passed through the base model, which is parameterised by W and colored green. The goal of the UnsupervisedUpdate is to modify W to achieve a top layer representation x L which performs well at few-shot learning. In order to train the base model, information is propagated backwards by the UnsupervisedUpdate in a manner analogous to backprop. Unlike in backprop however, the backward weights V are decoupled from the forward weights W. Additionally, unlike backprop, there is no explicit error signal as there is no loss. Instead at each layer, and for each neuron, a learning signal is injected by a meta-learned MLP parameterized by θ, with hidden state h. Weight updates are again analogous to those in backprop, and depend on the hidden state of the pre-and postsynaptic neurons for each weight. To allow easy comparison against other existing approaches, we present a more extensive survey of previous work in meta-learning in table form in TAB1, highlighting differences in choice of task, structure of the meta-learning problem, choice of meta-architecture, and choice of domain. To our knowledge, we are the first meta-learning approach to tackle the problem of unsupervised representation learning, where the inner loop consists of unsupervised learning. This contrasts with transfer learning, where a neural network is instead trained on a similar dataset, and then fine tuned or otherwise post-processed on the target dataset. We additionally believe we are the first representation meta-learning approach to generalize across input data modalities as well as datasets, the first to generalize across permutation of the input dimensions, and the first to generalize across neural network architectures (e.g. layer width, network depth, activation function). We consider a multilayer perceptron (MLP) with parameters φ t as the base model. The inner loop of our meta-learning process trains this base model via iterative application of our learned update rule. See Figure 1 for a schematic illustration and Appendix A for a more detailed diagram. In standard supervised learning, the'learned' optimizer is stochastic gradient descent (SGD). A supervised loss l (x, y) is associated with this model, where x is a minibatch of inputs, and y are the corresponding labels. The parameters φ t of the base model are then updated iteratively by performing SGD using the gradient ∂l(x,y) ∂φt. This supervised update rule can be written as φ t+1 = SupervisedUpdate(φ t, x t, y t ; θ), where t denotes the inner-loop iteration or step. Here θ are the meta-parameters of the optimizer, which consist of hyper-parameters such as learning rate and momentum. In this work, our learned update is a parametric function which does not depend on label information, φ t+1 = UnsupervisedUpdate(φ t, x t ; θ). This form of the update rule is general, it encompasses many unsupervised learning algorithms and all methods in Section 2.1.In traditional learning algorithms, expert knowledge or a simple hyper-parameter search determines θ, which consists of a handful of meta-parameters such as learning rate and regularization constants. In contrast, our update rule will have orders of magnitude more meta-parameters, including the weights of a neural network. We train these meta-parameters by performing SGD on the sum of the MetaObjective over the course of (inner loop) training in order to find optimal parameters θ *, DISPLAYFORM0 that minimize the meta-objective over a distribution of training tasks. Note that φ t is a function of θ since θ affects the optimization trajectory. In the following sections, we briefly review the main components of this model: the base model, the UnsupervisedUpdate, and the MetaObjective. See the Appendix for a complete specification. Additionally, code and meta-trained parameters θ for our meta-learned UnsupervisedUpdate is available 1. Our base model consists of a standard fully connected multi-layer perceptron (MLP), with batch normalization BID6, and ReLU nonlinearities. We chose this as opposed to a convolutional model to limit the inductive bias of convolutions in favor of learned behavior from the UnsupervisedUpdate. We call the pre-nonlinearity activations z 1, · · ·, z L, and post-nonlinearity activations DISPLAYFORM0 where L is the total number of layers, and x 0 ≡ x is the network input (raw data). The parameters are DISPLAYFORM1 where W l and b l are the weights and biases (applied after batch norm) for layer l, and V l are the corresponding weights used in the backward pass. We wish for our update rule to generalize across architectures with different widths, depths, or even network topologies. To achieve this, we design our update rule to be neuron-local, so that updates are a function of pre-and post-synaptic neurons in the base model, and are defined for any base model architecture. This has the added benefit that it makes the weight updates more similar to synaptic updates in biological neurons, which depend almost exclusively on local pre-and post-synaptic neuronal activity BID7. In practice, we relax this constraint and incorporate some cross neuron information to decorrelate neurons (see Appendix G.5 for more information).To build these updates, each neuron i in every layer l in the base model has an MLP, referred to as an update network, associated with it, with output h DISPLAYFORM0; θ where b indexes the training minibatch. The inputs to the MLP are the feedforward activations (x l & z l) defined above, and feedback weights and an error signal (V l and δ l, respectively) which are defined below. All update networks share meta-parameters θ. Evaluating the statistics of unit activation over a batch of data has proven helpful in supervised learning BID6. It has similarly proven helpful in hand-designed unsupervised learning rules, such as sparse coding and clustering. We therefore allow h l bi to accumulate statistics across examples in each training minibatch. During an unsupervised training step, the base model is first run in a standard feed-forward fashion, populating x l bi, z l bi. As in supervised learning, an error signal δ l bi is then propagated backwards through the network. Unlike in supervised backprop, however, this error signal is generated by the corresponding update network for each unit. It is read out by linear projection of the per-neuron hidden state h, δ l bi = lin h l bi, and propogated backward using a set of learned'backward weights' (V l) T, rather than the transpose of the forward weights (W l) T as would be the case in backprop (diagrammed in Figure 1). This is done to be more biologically plausible.Again as in supervised learning, the weight updates (∆W l) are a product of pre-and post-synaptic signals. Unlike in supervised learning however, these signals are generated using the per-neuron update networks: DISPLAYFORM1 bj, W ij. The full weight update (which involves normalization and decorrelation across neurons) is defined in Appendix G.5. The meta-objective determines the quality of the unsupervised representations. In order to meta-train via SGD, this loss must be differentiable. The meta-objective we use in this work is based on fitting a linear regression to labeled examples with a small number of data points. In order to encourage the learning of features that generalize well, we estimate the linear regression weights on one minibatch {x a, y a} of K data points, and evaluate the classification performance on a second minibatch {x b, y b} also with K datapoints, DISPLAYFORM0 are features extracted from the base model on data x a, x b, respectively. The target labels y a, y b consist of one hot encoded labels and potentially also regression targets from data augmentation (e.g. rotation angle, see Section 4.2). We found that using a cosine distance, CosDist, rather than unnormalized squared error improved stability. Note this meta-objective is only used during meta-training and not used when applying the learned update rule. The inner loop computation is performed without labels via the UnsupervisedUpdate. We choose to meta-optimize via SGD as opposed to reinforcement learning or other black box methods, due to the superior convergence properties of SGD in high dimensions, and the high dimensional nature of θ. Training and computing derivatives through long recurrent computation of this form is notoriously difficult BID9. To improve stability and reduce the computational cost we approximate the gradients DISPLAYFORM0 via truncated backprop through time BID10. Many additional design choices were also crucial to achieving stability and convergence in meta-learning, including the use of batch norm, and restricting the norm of the UnsupervisedUpdate update step (a full discussion of these and other choices is in Appendix B). Generalization in our learned optimizer comes from both the form of the UnsupervisedUpdate (Section 3.2), and from the meta-training distribution. Our meta-training distribution is composed of both datasets and base model architectures. We construct a set of training tasks consisting of CIFAR10 BID11 ) and multi-class classification from subsets of classes from Imagenet BID12 as well as from a dataset consisting of rendered fonts (Appendix H.1.1). We find that increased training dataset variation actually improves the meta-optimization process. To reduce computation we restrict the input data to 16x16 pixels or less during meta-training, and resize all datasets accordingly. For evaluation, we use MNIST BID13, Fashion MNIST BID14, IMDB BID15, and a hold-out set of Imagenet classes. We additionally sample the base model architecture. We sample number of layers uniformly between 2-5 and the number of units per layer logarithmically between 64 to 512.As part of preprocessing, we permute all inputs along the feature dimension, so that the UnsupervisedUpdate must learn a permutation invariant learning rule. Unlike other work, we focus explicitly on learning a learning algorithm as opposed to the discovery of fixed feature extractors that generalize across similar tasks. This makes the learning task much harder, as the UnsupervisedUpdate has to discover the relationship between pixels based solely on their joint statistics, and cannot "cheat" and memorize pixel identity. To provide further dataset variation, we additionally augment the data with shifts, rotations, and noise. We add these augmentation coefficients as additional regression targets for the meta-objective-e.g. rotate the image and predict the rotation angle as well as the image class. For additional details, see Appendix H.1.1. We implement the above models in distributed TensorFlow BID16. Training uses 512 workers, each of which performs a sequence of partial unrolls of the inner loop UnsupervisedUpdate, and computes gradients of the meta-objective asynchronously. Training takes ∼8 days, and consists of ∼200 thousand updates to θ with minibatch size 256. Additional details are in Appendix C. First, we examine limitations of existing unsupervised and meta learning methods. Then, we show meta-training and generalization properties of our learned optimizer and finally we conclude by visualizing how our learned update rule works. For details of the experimental setup, see Appendix H. To illustrate the negative consequences of objective function mismatch in unsupervised learnin algorithms, we train a variational autoencoder on 16x16 CIFAR10. Over the course of training we evaluate classification performance from few shot classification using the learned latent representations. Training curves can be seen in FIG0. Despite continuing to improve the VAE objective throughout training (not shown here), the classification accuracy decreases sharply later in training. To demonstrate the reduced generalization that from learning transferable features rather than an update algorithm, we train a prototypical network with and without the input shuffling described in Section 4.2. As the prototypical network primarily learns transferrable features, performance is significantly hampered by input shuffling. Results are in Continuing to optimize a variational auto-encoder (VAE) hurts few-shot accuracy after some number of steps (dashed line). Right: Prototypical networks transfer features rather than a learning algorithm, and perform poorly if tasks don't have consistent data structure. Training a prototypical network with a fully connected architecture (same as our base model) on a MiniImagenet 10-way classification task with either intact inputs (light purple) or by permuting the pixels before every training and testing task (dark purple). Performance with permuted inputs is greatly reduced (gray line). Our performance is invariant to pixel permutation. While training, we monitor a rolling average of the meta-objective averaged across all datasets, model architectures, and the number of unrolling steps performed. In FIG2 the training loss is continuing to decrease after 200 hours of training, which suggests that the approximate training techniques still produce effective learning. In addition to this global number, we measure performance obtained by rolling out the UnsupervisedUpdate on various meta-training and meta-testing datasets. We see that on held out image datasets, such as MNIST and Fashion Mnist, the evaluation loss is still decreasing. However, for datasets in a different domain, such as IMDB sentiment prediction BID15, we start to see meta-overfitting. For all remaining experimental , unless otherwise stated, we use meta-parameters, θ, for the UnsupervisedUpdate ing from 200 hours of meta-training. The goal of this work is to learn a general purpose unsupervised representation learning algorithm. As such, this algorithm must be able to generalize across a wide range of scenarios, including tasks that are not sampled i.i.d. from the meta-training distribution. In the following sections, we explore a subset of the factors we seek to generalize over. In FIG3, we compare performance on few shot classification with 10 examples per class. We evaluate test performance on holdout datasets of MNIST and Fashion MNIST at 2 resolutions: 14×14 and 28×28 (larger than any dataset experienced in meta-training). On the same base model architecture, our learned UnsupervisedUpdate leads to performance better than a variational autoencoder, supervised learning on the labeled examples, and random initialization with trained readout layer. The learned UnsupervisedUpdate generalizes to unseen datasets. Our learned update rule produces representations more suitable for few shot classification than those from random initialization or a variational autoecoder and outperforms fully supervised learning on the same labeled examples. Error bars show standard error. Right: Early in meta-training (purple), the UnsupervisedUpdate is able to learn useful features on a 2 way text classification data set, IMDB, despite being meta-trained only from image datasets. Later in meta-training (red) performance drops due to the domain mismatch. We show inner-loop training, consisting of 5k applications of the UnsupervisedUpdate evaluating the MetaObjective each iteration. Error bars show standard error across 10 runs. To further explore generalization limits, we test our learned optimizer on data from a vastly different domain. We train on a binary text classification dataset: IMDB movie reviews BID15, encoded by computing a bag of words with 1K words. We evaluate using a model 30 hours and 200 hours into meta-training (see FIG3 . Despite being trained exclusively on image datasets, the 30 hour learned optimizer improves upon the random initialization by almost 10%. When meta-training for longer, however, the learned optimizer "meta-overfits" to the image domain ing in poor performance. This performance is quite low in an absolute sense, for this task. Nevertheless, we find this very exciting as we are unaware of any work showing this kind of transfer of learned rules from images to text. We train models of varying depths and unit counts with our learned optimizer and compare at different points in training ( FIG4). We find that despite only training on networks with 2 to 5 layers and 64 to 512 units per layer, the learned rule generalizes to 11 layers and 10,000 units per layer. ReLU Leaky ReLU ELU Softplus Swish Tanh The learned UnsupervisedUpdate is capable of optimizing base models with hidden sizes and depths outside the meta-training regime. As we increase the number of units per layer, the learned model can make use of this additional capacity despite never having experienced it during meta-training. Right: The learned UnsupervisedUpdate generalizes across many different activation functions not seen in training. We show accuracy over the course of training on 14x14 MNIST.Next we look at generalization over different activation functions. We apply our learned optimizer on base models with a variety of different activation functions. Performance evaluated at different points in training (FIG4). Despite training only on ReLU activations, our learned optimizer is able to improve on random initializations in all cases. For certain activations, leaky ReLU BID17 and Swish BID18, there is little to no decrease in performance. Another interesting case is the step activation function. These activations are traditionally challenging to train as there is no useful gradient signal. Despite this, our learned UnsupervisedUpdate is capable of optimizing as it does not use base model gradients, and achieves performance double that of random initialization. To analyze how our learned optimizer functions, we analyze the first layer filters over the course of meta-training. Despite the permutation invariant nature of our data (enforced by shuffling input image pixels before each unsupervised training run), the base model learns features such as those shown in Figure 6, which appear template-like for MNIST, and local-feature-like for CIFAR10. Early in training, there are coarse features, and a lot of noise. As the meta-training progresses, more interesting and local features emerge. In an effort to understand what our algorithm learns to do, we fed it data from the two moons dataset. We find that despite being a 2D dataset, dissimilar from the image datasets used in meta-training, the learned model is still capable of manipulating and partially separating the data manifold in a purely unsupervised manner (Figure 6). We also find that almost all the variance in the embedding space is dominated by a few dimensions. As a comparison, we do the same analysis on MNIST. In this setting, the explained variance is spread out over more of the principal components. This makes sense as the generative process contains many more latent dimensions -at least enough to express the 10 digits. Figure 6: Left: From left to right we show first layer base model receptive fields produced by our learned UnsupervisedUpdate rule over the course of meta-training. Each pane consists of first layer filters extracted from φ after 10k applications of UnsupervisedUpdate on MNIST (top) and CIFAR10 (bottom). For MNIST, the optimizer learns image-template-like features. For CIFAR10, low frequency features evolve into higher frequency and more spatially localized features. For more filters, see Appendix D. Center: Visualization of learned representations before (left) and after (right) training a base model with our learned UnsupervisedUpdate for two moons (top) and MNIST (bottom). The UnsupervisedUpdate is capable of manipulating the data manifold, without access to labels, to separate the data classes. Visualization shows a projection of the 32-dimensional representation of the base network onto the top three principal components. Right: Cumulative variance explained using principal components analysis (PCA) on the learned representations. The representation for two moons data (red) is much lower dimensional than MNIST (blue), although both occupy a fraction of the full 32-dimensional space. In this work we meta-learn an unsupervised representation learning update rule. We show performance that matches or exceeds existing unsupervised learning on held out tasks. Additionally, the update rule can train models of varying widths, depths, and activation functions. More broadly, we demonstrate an application of meta-learning for learning complex optimization tasks where no objective is explicitly defined. Analogously to how increased data and compute have powered supervised learning, we believe this work is a proof of principle that the same can be done with algorithm design-replacing hand designed techniques with architectures designed for learning and learned from data via metalearning. Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gecsei. On the optimization of a synaptic learning rule. In Figure App.1: Schematic for meta-learning an unsupervised learning algorithm. We show the hierarchical nature of both the meta-training procedure and update rule. a) Meta-training, where the meta-parameters, θ, are updated via our meta-optimizer (SGD). b) The gradients of the MetaObjective with respect to θ are computed by backpropagation through the unrolled application of the UnsupervisedUpdate. c) UnsupervisedUpdate updates the base model parameters (φ) using a minibatch of unlabeled data. d) Each application of UnsupervisedUpdate involves computing a forward and "backward" pass through the base model. The base model itself is a fully connected network producing hidden states x l for each layer l. The "backward" pass through the base model uses an error signal from the layer above, δ, which is generated by a meta-learned function. e.) The weight updates ∆φ are computed using a convolutional network, using δ and x from the pre-and post-synaptic neurons, along with several other terms discussed in the text. Training and computing derivatives through recurrent computation of this form is notoriously difficult BID9. Training parameters of recurrent systems in general can lead to chaos. We used the usual techniques such as gradient clipping BID19, small learning rates, and adaptive learning rate methods (in our case Adam ), but in practice this was not enough to train most UnsupervisedUpdate architectures. In this section we address other techniques needed for stable convergence. When training with truncated backprop the problem shifts from pure optimization to something more like optimizing on a Markov Decision Process where the state space is the base-model weights, φ, and the'policy' is the learned optimizer. While traversing these states, the policy is constantly meta-optimized and changing, thus changing the distribution of states the optimizer reaches. This type of non-i.i.d training has been discussed at great length with respect to on and off-policy RL training algorithms BID21. Other works cast optimizer meta-learning as RL BID2 for this very reason, at a large cost in terms of gradient variance. In this work, we partially address this issue by training a large number of workers in parallel, to always maintain a diverse set of states when computing gradients. For similar reasons, the number of steps per truncation, and the total number of unsupervised training steps, are both sampled in an attempt to limit the bias introduced by truncation. We found restricting the maximum inner loop step size to be crucial for stability. BID22 studied the effect of learning rates with respect to the stability of optimization and showed that as the learning rate increases gradients become chaotic. This effect was later demonstrated with respect to neural network training in. If learning rates are not constrained, we found that they rapidly grew and entered this chaotic regime. Another technique we found useful in addressing these problems is the use of batch norm in both the base model and in the UnsupervisedUpdate rule. Multi-layer perceptron training traditionally requires very precise weight initialization for learning to occur. Poorly scaled initialization can make learning impossible BID23. When applying a learned optimizer, especially early in meta-training of the learned optimizer, it is very easy for the learned optimizer to cause high variance weights in the base model, after which recovery is difficult. Batch norm helps solve this issues by making more of the weight space usable. We implement the described models in distributed Tensorflow BID16. We construct a cluster of 512 workers, each of which computes gradients of the meta-objective asynchronously. Each worker trains on one task by first sampling a dataset, architecture, and a number of training steps. Next, each worker samples k unrolling steps, does k applications of the UnsupervisedUpdate(·; θ), computes the MetaObjective on each new state, computes DISPLAYFORM0 and sends this gradient to a parameter server. The final base-model state, φ, is then used as the starting point for the next unroll until the specified number of steps is reached. These gradients from different workers are batched and θ is updated with asynchronous SGD. By batching gradients as workers complete unrolls, we eliminate most gradient staleness while retaining the compute efficiency of asynchronous workers, especially given heterogeneous workloads which arise from dataset and model size variation. An overview of our training can be seen in algorithm F. Due to the small base models and the sequential nature of our compute workloads, we use multi core CPUs as opposed to GPUs. Training occurs over the course of ∼8 days with ∼200 thousand updates to θ with minibatch size 256. In this section we describe the details of our base model and learned optimizer. First, we describe the inner loop, and then we describe the meta-training procedure in more depth. In the following sections, λ name denotes a hyper-parameter for the learned update rule. The design goals of this system are stated in the text body. At the time of writing, we were unaware of any similar systems, so as such the space of possible architectures was massive. Future work consists of simplifying and improving abstractions around algorithmic components. An open source implementation of the UnsupervisedUpdate can be found at https: //github.com/tensorflow/models/tree/master/research/learning_ unsupervised_learning. The inner loop computation consists of iterative application of the UnsupervisedUpdate on a Base Model parameterized by φ, DISPLAYFORM0 where φ consists of forward weights, W l, biases, b l, parameterizing a multi-layer perceptron as well as backward weights, V l used by UnsupervisedUpdate when inner-loop training. This computation can be broken down further as a forward pass on an unlabeled batch of data, DISPLAYFORM1 where z l, and x l are the pre-and post-activations on layer l, and x 0 is input data. We then compute weight updates: DISPLAYFORM2 Finally, the next set of φ forward weights (W l), biases (b l), and backward weights (V l), are computed. We use an SGD like update but with the addition of a decay term. Equivalently, this can be seen as setting the weights and biases to an exponential moving average of the ∆W l, ∆V l, and ∆b l terms. DISPLAYFORM3 We use λ φlr = 3e − 4 in our work. ∆W l, ∆V l, ∆b l are computed via meta-learned functions (parameterized by θ).In the following sections, we describe the functional form of the base model, f, as well as the functional form of ComputeDeltaWeight (·; θ). The base model, the model our learned update rule is training, is an L layer multi layer perception with batch norm. We define φ as this model's parameters, consisting of weights (W l) and biases (b l) as well as the backward weights (V l) used only during inner-loop training (applications of UnsupervisedUpdate). We define N 1.. N L to be the sizes of the layers of the base model and N 0 to be the size of the input data. The forward computation parameterized by φ, consumes batches of unlabeled data from a dataset D: DISPLAYFORM0 DISPLAYFORM1 We define f (x; φ) as a function that returns the set of internal pre-and post-activation function hidden states as well asf (x; φ) as the function returning the final hidden state: DISPLAYFORM2 We define the MetaObjective to be a few shot linear regression. To increase stability and avoid undesirable loss landscapes, we additionally center, as well as normalize the predicted target before doing the loss computation. The full computation is as follows: DISPLAYFORM3 where N classes is the number of classes, and y, y are one hot encoded labels. First, the inputs are converted to embeddings with the base model, DISPLAYFORM4 x L =f (x ; φ) (App.19) Next, we center and normalize the prediction targets. We show this for y, but y is processed identically.ȳ DISPLAYFORM5 We then solve for the linear regression weights in closed form with features: x L and targets:ŷ. We account for a bias by concatenating a 1's vector to the features. DISPLAYFORM6 We then use these inferred regression weights C to make a prediction on the second batch of data, normalize the ing prediction, and compute a final loss, DISPLAYFORM7 Note that due to the normalization of predictions and targets, this corresponds to a cosine distance (up to an offset and multiplicative factor). The learned update rule is parameterized by θ. In the following section, we denote all instances of θ with a subscript to be separate named learnable parameters of the UnsupervisedUpdate. θ is shared across all instantiations of the unsupervised learning rule (shared across layers). It is this weight sharing that lets us generalize across different network architectures. The computation is split into a few main components: first, there is the forward pass, defined in f (x; φ). Next, there is a "backward" pass, which operates on the hidden states in a reverse order to propagate an error signal δ l back down the network. In this process, a hidden state h l is created for each layer. These h are tensors with a batch, neuron and feature index: h l ∈ R B,N l,λ hdims where λ hdims = 64. Weight updates to W l and b l are then readout from these h l and other signals found both locally, and in aggregate along both batch and neuron. In backprop, there exists a single scalar error signal that gets backpropagated back down the network. In our work, this error signal does not exist as there is no loss being optimized. Instead we have a learned top-down signal, dL, at the top of the network. Because we are no longer restricted to the form of backprop, we make this quantity a vector for each unit in the network, rather than a scalar, DISPLAYFORM0 In this work we set λ deltadims = 32. The architecture of TopD is a neural network that operates along every dimension of x L. The specification can be found in G.7.We structure our error propagation similarly to the structure of the backpropagated error signal in a standard MLP with contributions to the error at every layer in the network, DISPLAYFORM1 where δ l has the same shape as d l, and θ errorP ropW ∈ R λ hdims,λ deltadims and θ errorP ropB ∈ R λ deltadims. Note both θ errorP ropW and θ errorP ropB are shared for all layers l. In a similar way to backprop, we move this signal down the network via multiplying by a backward weight matrix (V l). We do not use the previous weight matrix transpose as done in backprop, instead we learn a separate set of weights that are not tied to the forward weights and updated along with the forward weights as described in G.5. Additionally, we normalize the signal to have fixed second moment,d DISPLAYFORM2 The internal h l ∈ R B,N l,λ hdims vectors are computed via: DISPLAYFORM3 The architecture of ComputeH is a neural network that operates on every dimension of all the inputs. It can be found in G.8. These definitions are recursive, and are computed in order: DISPLAYFORM4 With these computed, weight updates can be read out (Section G.5). When the corresponding symbols are not defined (e.g. z 0) a zeros tensor with the correct shape is used instead. The following is the implementation of ComputeDeltaWeight(DISPLAYFORM0 For a given layer, l, our weight updates are a mixture of multiple low rank readouts from h l and h l−1 . These terms are then added together with a learnable weight, in θ to form a final update. The final update is then normalized mixed with the previous weights. We update both the forward, W l, and the backward, V l, using the same update rule parameters θ. We show the forward weight update rule here, and drop the backward for brevity. For convenience, we define a low rank readout function LowRR that takes h l like tensors, and outputs a single lower rank tensor. DISPLAYFORM1 Here, Θ, is a placehoder for the parameters of the given readout. LowRR is defined as: DISPLAYFORM2 where λ gradc = 4 and is the rank of the readout matrix (per batch element). This sequence of terms allow the weight to be adjusted as a function of state in the pre-and postsynaptic neurons. They should be viewed as a basis function representation of the way in which the weight changes as a function of pre-and post-synaptic neurons, and the current weight value. We express each of these contributions to the weight update as a sequence of weight update planes, with the ith plane written ∆W DISPLAYFORM0 Each of these planes will be linearly summed, with coefficients generated as described in Equation App.57, in order to generate the eventual weight update. Ŵ DISPLAYFORM1 Additional weight update planes are designed to aid units in remaining decorrelated from each other's activity, and in decorrelating their receptive fields. Without terms like this, a common failure mode DISPLAYFORM2 We then normalize, re-weight, and merge each of these weight update planes into a single term, which will be used in the weight update step, DISPLAYFORM0 where θ mergeW ∈ R 10 (as we have 10 input planes).To prevent pathologies during training, we perform two post processing steps to prevent the learned optimizer from cheating, and increasing its effective learning rate, leading to instability. We only allow updates which do not decrease the weight matrix magnitude, DISPLAYFORM1 whereŴ l is W l scaled to have unit norm, and we normalize the length of the update, DISPLAYFORM2 To compute changes in the biases, we do a readout from h l. We put some constraints on this update to prevent the biases from pushing all units into the linear regime, and minimizing learning. We found this to be a possible pathology. DISPLAYFORM3 where θ Breadout ∈ R λ hdims.We then normalize the update via the second moment: DISPLAYFORM4 Finally, we define ComputeDeltaWeight as all layer's forward weight updates, and backward weight updates, and bias updates. DISPLAYFORM5 This function performs various convolutions over the batch dimension and data dimension. For ease of notation, we use m as an intermediate variable. Additionally, we drop all convolution and batch norm parameterizations. They are all separate elements of θ topD. We define two 1D convolution operators that act on rank 3 tensors: ConvBatch which performs convolutions over the zeroth index, and ConvUnit which performs convolutions along the firs index. We define the S argument to be size of hidden units, and the K argument to be the kernel size of the 1D convolutions. Additionally, we set the second argument of BatchNorm to be the axis normalized over. DISPLAYFORM6 First, we reshape to add another dimension to the end of x L, in pseudocode: DISPLAYFORM7 Next, a convolution is performed on the batch dimension with a batch norm and a ReLU non linearity. This starts to pull information from around the batch into these channels. DISPLAYFORM8 We set λ topdeltasize = 64. Next, a series of unit convolutions (convolutions over the last dimension) are performed. These act as compute over information composed from the nearby elements of the batch. These unit dimensions effectively rewrite the batch dimension. This restricts the model to operate on a fixed size batch. DISPLAYFORM9 Next a series of 1D convolutions are performed over the batch dimension for more compute capacity. Finally, we convert the representations to the desired dimensions and output. DISPLAYFORM10 With the inputs prepared, we next perform a series of convolutions on batch and unit dimensions,. DISPLAYFORM11 The is then output. ComputeH (·; θ computeH) = m 11 (App.104)We set λ computehsize = 64 which is the inner computation size. DISPLAYFORM12 We trained on a data distribution consisting of tasks sampled uniformly over the following datasets. Half of our training tasks where constructed off of a dataset consisting of 1000 font rendered characters. We resized these to 14x14 black and white images. We call this the glyph dataset. "Alphabet" is an example of such a dataset consisting of alphabet characters. We used a mixture 10, 13, 14, 17, 20, and 30 way classification problems randomly sampled, as well as sampling from three 10-way classification problems sampled from specific types of images: letters of the alphabet, math symbols, and currency symbols. For half of the random sampling and all of the specialized selection we apply additional augmentation. This augmentation consists of random rotations (up to 360 degrees) and shifts up to +-5 pixels in the x and y directions. The parameters of the augmentation were inserted into the regression target of the MetaObjective as a curriculum of sorts and to provide diverse training signal. In addition to the the glyph set, we additionally used Cifar10, resized to 16x16, as well as 10, 15, 20, and 25 way classification problems from imagenet. Once again we resized to 16x16 for compute reasons. With a dataset selected, we apply additional augmentation with some probability consisting of the following augmentations. A per task dropout mask (fixed mask across all images in that task). A per example dropout mask (a random mask per image). A permutation sampled from a fixed number of pre created permutations per class. A per image random shift in the x direction each of image. All of these additional augmentations help with larger domain transfer. We employ Adam as our meta-optimizer. We use a learning rate schedule of 3e-4 for the first 100k steps, then 1e-4 for next 50k steps, then 2e-5 for remainder of meta-training. We use gradient clipping of norm 5 on minibatchs of size 256.We compute our meta-objective by averaging 5 evaluation of the linear regression. We use a ridge penalty of 0.1 for all this work. When computing truncated gradients, we initially sample the number of unrolled applications of the UnsupervisedUpdate in a uniform distribution of. This is the number of steps gradients are backpropogated through. Over the course of 50k meta-training steps we uniformly increase this to. This increases meta-training speed and stability as large unrolls early in training can be unstable and don't seem to provide any value. For sampling the number of truncated steps (number of times the above unrolls are performed), we use a shifted normal distribution -a normal distribution with the same mean and standard deviation. We chose this based on the expected distribution of the training step, φ iteration number, across the cluster of workers. We initially set the standard deviation low, 20, but slowly increased it over the course of 5000 steps to 20k steps. This slow increase also improved stability and training speed. For each experimental figure, we document the details. The VAE we used consists of 3 layers, size 128, with ReLU activations and batch norm between each layer. We then learn a projection to mean and log std of size 32. We sample, and use the inverse architecture to decode back to images. We use a quantized normal distribution (once again parameterized as mean and log std) as a posterior. We train with Adam with a learning rate of 1e-4.To isolate the effects of objective function mismatch and overfitting, we both train on the unlabeled training set and evaluate on the labeled training set instead of a validation set. We use a 4 layer, size 128 unit architecture with a 32 layer embedding for all models. We select performance at 100k training steps for the VAE, and 3k for our learned optimizer. Our supervised learning baseline consists of the same architecture for the base model but with an additional layer that outputs log probabilities. We train with cross entropy loss on a dataset consisting of only 10 examples per class (to match the other numbers). Surprisingly, the noise from batch norm acts as a regularizer, and allowing us to avoid needing a complex early stopping scheme as test set performance simply plateaus over the course of 10k steps. We train with Adam with a learning rate of 3e-3, selected via a grid over learning rate on test set performance. In this setting, having a true validation set would dramatically lower the amount of labeled data available (only 100 labeled examples) and using the test set only aids in the this baseline's performance. For the IMDB experiments, we tokenized and selected the top 1K words with an additional set of tokens for unknown, sentence start, and sentence end. Our encoding consisted of a 1 if the word is present, otherwise 0. We used the same 4 layer, 128 hidden unit MLP with an addition layer outputting a 32 dimensional embedding. We used ReLU activations and 4 layers of size 128 with an additional layer to 32 units unless otherwise specified by the specific experiment. We trained prototypical networks on either intact or shuffled mini-Imagenet images. For shuffled images, we generated a fixed random permutation for the inputs independently for every instantiation of the base network (or 'episode' in the meta-learning literature ). The purpose of shuffling was to demonstrate the inductive bias of this type of meta-learning, namely that they do not generalize across data domain. Note that the base network trained was the same fully connected architecture like that used in this paper (3 layers, size 128, with ReLU activations and batch normalization between layers). Though the original paper used a convolutional architecture, here we swapped it with the fully connected architecture because the tied weights in a convolutional model do not make sense with shuffled pixels.

We learn an unsupervised learning algorithm that produces useful representations from a set of supervised tasks. At test-time, we apply this algorithm to new tasks without any supervision and show performance comparable to a VAE.

Represent the short summary for finding the corresponding scientific article (long).

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.


Question: Represent the article for finding a single-sentence TLDR.

Question: There is significant recent evidence in supervised learning that, in the over-parametrized setting, wider networks achieve better test error. In other words, the bias-variance tradeoff is not directly observable when increasing network width arbitrarily. We investigate whether a corresponding phenomenon is present in reinforcement learning. We experiment on four OpenAI Gym environments, increasing the width of the value and policy networks beyond their prescribed values. Our empirical lend support to this hypothesis. However, tuning the hyperparameters of each network width separately remains as important future work in environments/algorithms where the optimal hyperparameters vary noticably across widths, confounding the when the same hyperparameters are used for all widths. A longstanding notion in supervised learning is that, as model complexity increases, test error decreases initially and, eventually, increases again. Intuitively, the idea is that as the size of your hypothesis class grows, the closer you can approximate the ground-truth function with some function in your hypothesis class. At the same time, the larger amount of functions to choose from in your hypothesis class leads to higher estimation error (overfitting) from fitting the finite data sample too closely. This is the essential bias-variance tradeoff in supervised learning. We discuss these tradeoffs in more depth in Section 2.2.However, BID20 found that increasing the width of a single hidden layer neural network leads to decreasing test error on MNIST and CIFAR-10. Since then, there has been a large amount of evidence that wider networks generalize better in a variety of different architectures and hyperparameter settings BID27 BID21 BID15 BID19 BID0 BID24 BID17, once in the over-parametrized setting BID24 BID0. In other words, the biasvariance tradeoff is not observed in this over-parametrized setting, as network width grows BID19.How far can we inductively infer from this? Is this phenomenon also present in deep reinforcement learning or do we eventually see a degradation in performance as we increase network width? In this paper, we present preliminary evidence that this phenomenon is also present in reinforcement learning. For example, using default hyperparameters, we can already see performance increase well past the default width that is commonly used in FIG0. We test the hypothesis that wider networks (both policy and value) perform monotonically better than their smaller counterparts in policy gradients methods. Of course, we will hit diminishing returns as the network width gets very large, but this is very different from the competing hypothesis that larger networks will overfit more. We are given a training set S = {(x 1, y 1), (x 2, y 2),..., (x m, y m)} of m training examples, where x i ∈ X and y i ∈ Y. Furthermore, Z = X × Y, so S ∈ Z m. D denotes a distribution over Z, so we have (x i, y i) ∼ D and S ∼ D m. We use lowercase x and y to denote random variables because of convention in this field. We learn a hypothesis h ∈ H via a learning algorithm A: Z m → H. We denote a hypothesis learned from training set S as h S = A(S). Given a loss function,: Y ×Y → R, the goal is to minimize the expected risk: DISPLAYFORM0 We present a discussion on tradeoffs in model complexity because it does not appear to be much of a focus in the reinforcement learning community. A common way of thinking about the generalization performance of a learner is through the lens of a tradeoff. For example, when h S is chosen from a hypothesis class H, R(h S) can be decomposed into approximation error and estimation error DISPLAYFORM0 where E app = min h∈H R(h) and E est = R(h S) − E app. Shalev-Shwartz & Ben-David (2014, Section 5.2) present this decomposition and frame it as a tradeoff. BID2 describe this as the "well known tradeoff between approximation error and estimation error" and present it in a slightly more lucid way as a decomposition of the excess risk: DISPLAYFORM1 where R(h *) is the Bayes error and h * H = arg min h∈H R(h) is the best hypothesis in H. The approximation error can then be interpreted as the distance of the best hypothesis in H from the Bayes classifier, and the estimation error can be interpreted as the average distance of the learned hypothesis from the best hypothesis in H. It is common to associate larger H with smaller approximation error and larger estimation error. The commonly cited universal approximation property of neural networks BID4 BID13 BID16 means that the approximation error goes to 0 as the network width increases; these do not say anything about estimation error. A similar tradeoff in model complexity is known as the bias-variance tradeoff BID7. Bias is analogous to the approximation error while variance is analogous to the estimation error. This tradeoff is probably even more pervasive (Bishop (2006, Chapter 3 .2), BID7, Hastie et al. (2001, Chapter 2.9), Goodfellow et al. (2016, Chapter 5.4.4) ). It is common to view the problem of designing a good learning algorithm as choosing a good H that optimizes this tradeoff. Statistical learning theory for supervised learning is given in the i.i.d. setting. That is, examples are independent and identically distributed. This also means the training distribution is the same as the test distribution. In reinforcement learning, training examples are not independent because examples within the same episode depend on each other through the current behavior policy and through the environment's transition dynamics. Training examples are not identically distributed because the policy produces training examples, and the policy changes over time. For the same reason, the training distribution and the test distribution are not completely the same. These differences make it nonobvious that the phenomenon seen in supervised learning would extend to reinforcement learning. We run experiments, with a variety of combinations of environments and learning algorithms, where we vary the width of the shared policy and value network. We use four different environments from OpenAI Gym BID3: CartPole, Acrobot, MountainCar, and Pendulum. We use four different learning algorithms: PPO, A2C BID18, ACER BID25, and ACKTR. We make use of the existing implementations of these algorithms in the Stable Baselines library BID12, an improved fork of OpenAI Baselines. We were only able to train ACKTR up to width 512 because it is an approximate second-order method. Experiments with ACKTR are in Appendix B.We get hyperparameters that were tuned on networks of width 64 from the RL Baselines Zoo that was built on top of Stable Baselines. One hyperparameter is how many time steps the learners are trained for. It is different for different environment/learner pairs, but always on the order of 1 million. It is always the same across widths within an environment/learner pair. In some of the plots, learners see fewer episodes because their episodes are, on average, longer. We choose relatively simple tasks for these experiments partially because they are faster to train on, but more importantly, we choose them because their simplicity lends itself to more ease of overfitting. In other words, on these tasks, we will see diminishing returns with much smaller networks, so we can test the "very wide networks will not see degraded performance" hypothesis with a much smaller range of networks. We run each experiment with 5 different random seeds. The policy and value networks are shared. The architecture consists of 2 hidden fully connected layers followed by separate linear transformations: one to yield the policy and one to yield the value. We use 2 hidden layers, rather than just 1, because 2 hidden layers are more common in reinforcement learning. In CartPole (Fig. 2), we see a lot of evidence for the hypothesis. In the both the PPO and A2C experiments, peak performance is reached by width 64, and that level of performance is maintained through width 2048. In the ACER experiment, near peak performance is reached by width 128, and through width 2048, we see peak performance. Similarly, in Acrobot (Fig. 3), we see even more evidence for the hypothesis. We see peak performance as early as width 16 in PPO, ACER, and A2C. This means that Acrobot is simple enough to only require a network of width 16 (compared to 64 for CartPole). Still, we see peak performance through width 2048 in all 3 learners. In Pendulum (Appendix A), we see more evidence for the hypothesis. The default width network, performs distinctly worse than the wider networks. We do not see any degradation of performance through width 2048. We only run PPO with the Pendulum environment because RL Baselines Zoo did not have tuned hyperparameters for the other algorithms. In the MountainCar environment, we see the first hint of what looks like could be evidence against the hypothesis (Fig. 4). PPO (left) performance begins to degrade at width 2048, ACER (center) performance begins to degrade at width 512, and we see a sharp drop in performance from width 1024 to width 2048 in A2C (right).RL algorithms are known to be highly sensitive to hyperparameter settings BID10 BID14, especially learning rate BID11. We believe this performance degradation is due to more variability across widths of the optimal hyperparameters on MountainCar (compared CartPole, Acrobot, and Pendulum). In order to fairly compare all the widths, we would like the hyperparameters for each of them to be optimal. BID6 ) study test error when scaling network width in supervised learning, and they scale the learning rate as h −1.5, where h is the network width. This scaling is motivated by making the number of steps to convergence independent of width, but it does not necessarily make the learning rate for each network optimal. Because learning rate is such an important and sensitive hyperparameter in reinforcement learning BID11, we try scaling the learning rate α with both of the following schemes: α ← min(α * 64, (h/64) −1 ) and α ← min(α * 64, (h/64) −1.5 ), where α * 64 is the learning rate that was tuned to network width 64 (pulled from RL Baselines Zoo). We see that scaling the learning rate as h −1 (Fig. 5) and h −1.5 (Appendix C, Fig. 8) actually make the largest networks perform worse, indicating that this scaling is not useful for comparing networks with optimal hyperparameters. We present these scalings on MountainCar because it was the environment that did not look like the others, but the scalings on CartPole and Acrobot are in Appendix D. The phenomenon in supervised learning that motivated this work is that, in the over-parametrized setting, increasing network width leads to monotonically lower test error (no U curve). We find a fair amount of evidence of this phenomenon extending to reinforcement learning in our preliminary experiments (namely CartPole, Acrobot, and Pendulum).However, we also saw that performance did consistently degrade in the MountainCar experiments. We believe this to be because that environment is more sensitive to hyperparameters; since the hyperparameters were chosen using width 64 and then used for all of the other widths, the hyperparameters are likely not optimal for the other widths like they are for width 64. The MountainCar environment exaggerates this lack suboptimality more than the other 3 environments. The main next experiments we plan to run will use an automated tuning procedure that chooses the hyperparameters for each width individually. We believe this protocol will yield MountainCar that look much more like the CartPole and Acrobot . We then plan to replicate these findings across more learning algorithms and more environments. A. Pendulum

Over-parametrization in width seems to help in deep reinforcement learning, just as it does in supervised learning.

Represent the short summary for finding the corresponding scientific article (long)

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Learning disentangled representations of data is one of the central themes in unsupervised learning in general and generative modelling in particular. In this work, we tackle a slightly more intricate scenario where the observations are generated from a conditional distribution of some known control variate and some latent noise variate. To this end, we present a hierarchical model and a training method (CZ-GEM) that leverages some of the recent developments in likelihood-based and likelihood-free generative models. We show that by formulation, CZ-GEM introduces the right inductive biases that ensure the disentanglement of the control from the noise variables, while also keeping the components of the control variate disentangled. This is achieved without compromising on the quality of the generated samples. Our approach is simple, general, and can be applied both in supervised and unsupervised settings. Consider the following scenario: a hunter-gatherer walking in the African Savannah some 50,000 years ago notices a lioness sprinting out of the bush towards her. In a split second, billions of photons reaching her retinas carrying an enormous amount of information: the shade of the lioness' fur, the angle of its tail, the appearance of every bush in her field of view, the mountains in the and the clouds in the sky. Yet at this point there is a very small number of attributes which are of importance: the type of the charging animal, its approximate velocity and its location. The rest are just details. The significance of the concept that the world, despite its complexity, can be described by a few explanatory factors of variation, while ignoring the small details, cannot be overestimated. In machine learning there is a large body of work aiming to extract low-dimensional, interpretable representations of complex, often visual, data. Interestingly, many of the works in this area are associated with developing generative models. The intuition is that if a model can generate a good approximation of the data then it must have learned something about its underlying representation. This representation can then be extracted either by directly inverting the generative process (b) or by extracting intermediate representations of the model itself . Clearly, just learning a representation, even if it is low-dimensional, is not enough. The reason is that while there could be many ways to compress the information captured in the data, allowing good enough approximations, there is no reason to a priori assume that such a representation is interpretable and disentangled in the sense that by manipulating certain dimensions of the representation one can control attributes of choice, say the pose of a face, while keeping other attributes unchanged. The large body of work on learning disentangled representations tackles this problem in several settings; fully supervised, weakly supervised and unsupervised, depending on the available data (; ; ; ; ; ; ;). Ideally, we would like to come up with an unsupervised generative model that can generate samples which approximate the data to a high level of accuracy while also giving rise to a disentangled and interpretable representation. In the last decade two main approaches have captured most of the attention; Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs). In their original versions, both GANs and VAEs were trained in an unsupervised manner and (a) Chair rotation generated by CGAN (b) Chair rotation generated by CZ-GEM Figure 1: Changing the azimuth of chairs in CGAN and CZ-GEM while holding Z constant. Unlike CZ-GEM, C and Z are clearly entangled in CGAN as changing C also changes the type of chair even though Z is held constant. gave rise to entangled representations. Over the years, many methods to improve the quality of the generated data as well as the disentanglement of the representations have been suggested (; ; ;). By and large, GANs are better than VAEs in the quality of the generated data while VAEs learn better disentangled representations, in particular in the unsupervised setting. In this paper, we present a framework for disentangling a small number of control variables from the rest of the latent space which accounts for all the additional details, while maintaining a high quality of the generated data. We do that by combining VAE and GAN approaches thus enjoying the best of both worlds. The framework is general and works in both the supervised and unsupervised settings. Let us start with the supervised case. We are provided with paired examples (x, c) where x is the observation and c is a control variate. Crucially, there exists a one-to-many map from c to the space of observations, and there are other unobserved attributes z (or noise) that together completely define x. For instance, if x were an image of a single object, c controls the orientation of the object relative to the camera and z could represent object identity, texture or . Our goal is to learn a generative model p θ (x|c, z) that fulfills two criteria: If we were learning models of images, we would like the generated images to look realistic and match the true conditional distribution p(x|c). The posterior is factorized p(c, z|x; θ) = p(c|x; θ)p(z|x; θ): We would like the control variate to be disentangled from the noise. For example, changing the orientation of the object should not change the identity under our model. This problem setup can occur under many situations such as learning approximate models of simulators, 3D reconstructions, speaker recognition (from speech), and even real-world data processing in the human brain as in the hunter-gatherer example above. We argue that a naive implementation of a graphical model as shown in Figure 2 (left), e.g. by a conditional GAN , does not satisfy Criterion 2. In this model, when we condition on x, due to d-separation, c and z could become dependent, unless additional constraints are posed on the model. This effect is demonstrated in Figure 1 (a). To overcome this we split the generative process into two stages by replacing C with a subgraph (C → Y) as shown in Figure 2 (center). First, we generate a crude approximation y of the data which only takes c into account. The is a blurry average of the data points conditioned on c, see Figure 2 (right). We then feed this crude approximation into a GAN-based generative model which adds the rest of the details conditioned on z. We call this framework CZ-GEM. The conditioning on z in the second stage must be done carefully to make sure that it does not get entangled with y. To that end we rely on architectural choices and normalization techniques from the style transfer literature 2. The is a model which generates images of high quality while disentangling c and z as can be clearly seen in Figure 1(b). Additionally, in the unsupervised setting, when the labels c are not available, (C → Y) can be realized by β-VAE, a regularized version of VAE which has been shown to learn a disentangled representation of its latent variables . In Section 3 we provide implementation details for both the supervised and unsupervised versions. We summarize our two main contributions: Figure 2: On the left, a conditional GAN (CGAN) model. CZ-GEM in the middle replaces node C with a subgraph (C → Y) that is trained independently of the rest of the model. This subgraph learns to only partially render the observation. As such, Z comes at a later stage of the rendering pipeline to add details to Y. As an example, consider the rightmost graph where the observation is made up of different types of chairs in different poses. Let the pose be controlled by C and the type (Identity) be explained by Z. Then in step one of CZ-GEM we learn the pose relationship between C and X via the subgraph, giving rise to a blurry chair in the correct pose. Once the pose is learned, in the second step, the approximate rendering Y is transformed into X by allowing Z to add identity related details to the blurry image. We break down the architecture to model an intermediate representation that lends itself to interpretability and disentanglement, and then (carefully) use a GAN based approach to add the rest of the details, thus enjoying a superior image generation quality compared to VAEs. We show that our model can be combined easily with common methods for discovering disentangled representations such as β-VAE to extract c and treat them as labels to generate images that do not compromise on generative quality. Generative adversarial networks (GAN) represent the current state of the art in likelihood-free generative modeling. In GANs, a generator network G θ is trained to produce samples that can fool a discriminator network D ω that is in turn trained to distinguish samples from the true data distribution p(x) and the generated samples Here, p z is usually a low dimensional easy-to-sample distribution like standard Gaussian. A variety of tricks and techniques need to be employed to solve this min-max optimization problem. For our models, we employ architectural constraints proposed by DC-GAN that have been widely successful in ensuring training stability and improving generated image quality. Conditional GANs (CGAN) adapt the GAN framework for generating class conditional samples by jointly modeling the observations with their class labels. In CGAN, the generator network G θ is fed class labels c to produce fake conditional samples and the discriminator D ω is trained to discriminate between the samples from the joint distribution of true conditional and true labels p(x|c)p(c) and the fake conditional and true labels p θ (x|c)p(c). While not the main focus of this paper, we present a novel information theoretic perspective on CGANs. Specifically, we show that CGAN is trained to maximize a lower-bound to the mutual information between the observation and its label while simultaneously minimizing an upper-bound to it. We state this formally: by training a discriminator D ω to approximate the log-ratio of the true and generated data densities i.e. D ω ≈ log p(x, c)/p θ (x, c) in turn minimizing the following where I g,θ (x, c) is the generative mutual information and q(c|x, θ) is the posterior under the learned model. The detailed derivation is provided in Appendix A.1. Notice that at the limit, the model learns exactly the marginal distribution of x and the posterior q(c|x) and the KL terms vanish. Variational autoencoders (VAE) represent a class of likelihood-based deep generative models that have recently been extensively studied and used in representation learning tasks (; ; . Consider a latent variable model where observation X is assumed to be generated from some underlying low-dimensional latent feature space Z. VAE models learn the conditional distribution p(x|z) using a deep neural network (parameterized by θ) called decoder network. It uses another deep neural network (parameterized by φ), called encoder to model the posterior distribution p(z|x). The encoder and decoder networks are trained using amortized variational inference to maximizes a variational lower-bound to the evidence likelihood (ELBO). showed that by regularizing the variational posterior approximation of p(z|x) to be close to the prior distribution p(z) in KLdivergence, the model is encouraged to learn disentangled representations. I.e. the model learns a posterior distribution that is factorized over the dimensions. They call their model β-VAE. We note that information bottleneck based methods for disentangled representation learning, such as β-VAE, severely compromise the generative quality. Batch-Normalization (BN) plays a crucial role in ensuring the stability of GAN training. However, as we discuss in Section 3, it is not suitable for our purposes. Recently, it has been shown that Instance Normalization (IN) and its variant Adaptive Instance Normalization (AdaIN) can be particularly useful for image generation and stylization. IN normalizes each convolutional channel per training sample, while AdaIN modifies this normalization to be a function of an additional variable z (usually style in style transfer). The final transformation applied by AdaIN is: where µ(x) = 1 HW h,w x nhwc and σ(x) = 1 HW h,w (x nhwc − µ(x)) 2 +. γ(z) and β(z) are learned functions of z that could be parameterized by a neural network, usually a fully connected layer. In Section 1, we provided a high level description of our approach. We will now provide a detailed description of how the two components of CZ-GEM, subgraph C → Y and the conditional generative models (Y, Z) → X are implemented and trained in practice. Figure 3 provides an implementation schematic of our proposed framework. If C is known a priori then learning the subgraph C → Y reduces to the regression problem that minimizes ||x c − y c || 2. In practice, since our observations are images, this subgraph is realized using a deep transposed-convolution based decoder network and is trained to learn the map between C and Y. This is similar to the recent work of Srivastava et al. (2019b). We emphasize that this network is trained independently of the rest of the model. to discover disentangled generative control factors. In our implementation we use β-VAE (see Section 2.2). One drawback of these information bottleneck based methods is that they compromise on the generative quality. This is where the GAN likelihood-free approach in the second stage comes into play. In fact, even if the output of the first stage (i.e. the intermediate image Y in Figure 2) is of very low generative quality, the final image is of high quality since the second stage explicitly adds details using a state-of-the-art GAN method. In Section 5 we show how a simple VAE with a very narrow information bottleneck (2-6 dimensions) can be used within CZ-GEM to discover C in an unsupervised fashion without compromising on generation quality. Vanilla GANs can only model the marginal data distribution i.e. they learn p θ to match p x and in doing so they use the input to the generator (G θ) only as a source of stochasticity. Therefore we start with a conditional GAN model instead, to preserve the correspondence between Y and X. As shown in section 2.1, this framework trains G θ such that the observation X is maximally explained by the conditioning variable Y. One major deviation from the original model is that the conditioning variable in our case is the same type and dimensionality as the observation. That is, it is an image, albeit a blurry one. This setup has previously been used by in the context of image-to-image translation. Incorporating Z requires careful implementation due to two challenges. First, trivially adding Z to the input along with Y invokes d-separation and as a Y and Z can get entangled. Intuitively, Z is adding high level details to the intermediate representation Y. We leverage this insight as an inductive bias, by incorporating Z at higher layers of the network rather than just feeding it as an input to the bottom layer. A straightforward implementation of this idea does not work tough. The reason is that BatchNorm uses batch-level statistics to normalize the incoming activations of the previous layer to speed up learning. In practice, mini-batch statistics is used to approximate batch statistics. This adds internal stochasticity to the generator causing it to ignore any externally added noise, such as Z. An elegant solution to resolve this second challenge comes in the form of adaptive instance normalization (see Section 2.3). It not only removes any dependency on the batch-statistics but also allows for the incorporation of Z in the normalization process itself. For this reason, it has previously been used in style transfer tasks . We replace all instances of BatchNorm in the generator with Adaptive InstanceNorm. We then introduce Z to the generative process using equation 1. γ(z) and β(z) are parameterized as a simple feed-forward network and are applied to each layer of AdaIN in the generator. Disentangled representation learning has been widely studied in recent years, both in the supervised and unsupervised settings. In supervised cases, works such as has emphasized the use of inductive biases and weak supervision instead of fully unsupervised methods for disentangled representation learning. and have successfully shown that including inductive biases, respectively an explicit 3D representation, leads to better performance. Their inductive bias comes in the form of learned 3D transformation pipeline. In comparison, CZ-GEM is much simpler and smaller in design and applies to the general setting where the data is determined by control and noise variables. In addition, it can be used in both supervised and unsupervised setting and does not rely on the knowledge of 3D transformations. Manually disentangled generative models like the 3D morphable model have been built for faces. They are powerful in terms of generalization but there is a big gap between those synthetic images and real-world face images. In addition, those models are built highly supervised from 3D scans and the approach is limited by the correspondence assumption which does not scale to more complex objects like chairs. We use a 3D morphable model to generate our synthetic face dataset and show that we can disentangle pose variation from synthetic and real 2D images. In this section, we provide a comprehensive set of quantitative and qualitative to demonstrate how CZ-GEM is clearly able to not only disentangle C from Z in both supervised and unsupervised settings but also ensure that independent components of C stay disentangled after training. Additionally, we show how in unsupervised settings CZ-GEM can be used to discover disentangled latent factors when C is not explicitly provided. We evaluate CZ-GEM on a variety of image generation tasks which naturally involve observed attributes C and unobserved attributes Z. To that end, we generate three 3D image datasets of faces, chairs, and cars with explicit control variables. Chairs and cars datasets are derived from ShapeNet . We sample 100k images from the full yaw variation and a pitch variation of 90 degrees. We used the straight chair subcategory with 1968 different chairs and the sedan subcategory with 559 different cars. We used Blender to render the ShapeNet meshes scripted with the Stanford ShapeNet renderer. For faces, we generated 100k images from the Basel Face Model 2017 . We sample shape and color (first 50 coefficients), expressions (first 5 coefficients), pose (yaw -90 to 90 degrees uniformly, pitch and roll according to a Gaussian with variance of 5 degrees) and the illumination from the Basel Illumination Prior. For the generation of the faces dataset, we use the software provided by. For the stated datasets we have complete access to C, but we also include unsupervised on celebA with unconstrained real images. All our datasets are built from publicly available data and tools. We use the DCGAN architecture for all neural networks involved in all the experiments in this work and provide a reference implementation with exact architecture and hyperparameter settings at https://github.com/AnonymousAuthors000/CZ-GEM. In the supervised setting we compare CZ-GEM to CGAN. We quantitatively compare the two methods to ensure that independent components of C stay disentangled post learning. Furthermore, we qualitatively compare their abilities to disentangle C and Z. And finally, we compare the quality of the samples that the models generate. For chairs and cars, C contains only the pose variables and all other variations are explained by Z. For faces, C contains in addition to pose the first 4 principal directions of shape variations. three datasets. We also include the training error (i.e. the MSE of the regressor on the real data) for comparison. The show that CGAN and CZ-GEM are comparable in preserving the label information in the generated data, but as we show below, only CZ-GEM does that while ensuring that C and Z remain disentangled. To qualitatively evaluate the level of disentanglement between C and Z, we vary each individual dimension of C over its range while holding Z constant. We plot the generated images for both models on car and chair datasets in Figure 4. Notice that CZ-GEM allows us to vary the control variates without changing the identity of the object, whereas CGAN does not. In addition, we find that for CGAN, the noise Z provides little to no control over the identity of the chairs. This is potentially due to the internal stochasticity introduced by the BatchNorm. The last rows for the CZ-GEM figures provide the visualization of Y. It can be seen how Y is clearly preserving C (pose information) but averaging the identity related details. We also qualitatively evaluate CZ-GEM on the more challenging faces dataset that includes 10 control variates. As shown in Figure 9 in the appendix, CZ-GEM is not only able to model the common pose factors such as rotation and azimuth but also accurately captures the principal shape component of Basel face model that approximates the width of the forehead, the width of jaw etc. Compared to CGAN, CZ-GEM does a qualitatively better job at keeping the identity constant. Finally, in order to ensure that our method does not compromise the generative quality, we evaluate the Inception score on all three datasets. Inception score has been widely used to measure the diversity and the generative quality of GANs. As shown in Table 2, unlike CGAN, CZ-GEM does not degrade the image quality. We now test the performance of CZ-GEM in the unsupervised setting, where disentangled components of C needs to be discovered, using β-VAE, as part of learning the mapping C → Y. For our purpose, we use a simple version of the original β-VAE method with a very narrow bottleneck (6D for faces and 2D for cars and chairs) to extract C. The latent traversals for the faces dataset are presented in Figure 5. Unsupervised discovery is able to recover rotation as well as translation variation present in the dataset. For comparison, we evaluate InfoGAN and present the in Figure 6 where it is evident that CZ-GEM clearly outperforms InfoGAN on both disentanglement and generative quality. More traversal are provided in the appendix. We further test our method on the CelebA dataset , where pose information is not available. This traversal plot is shown in Figure 7. Traversal plots for cars and chairs dataset are provided in the Appendix Figure 12 and Figure 13. We present a simple yet effective method of learning representations in deep generative models in the setting where the observation is determined by control variate C and noise variate Z. Our method ensures that in the learned representation both C and Z are disentangled as well as the components of C themselves. This is done without compromising the quality of the generated samples. In future work, we would like to explore how this method can be applied to input with multiple objects. Apart from the MSE-based estimator reported in 1, we report and additional evaluation measure. We use the same regressor f (x) trained for 1, but we report the Pearson correlation co-efficient (r) between the predicted label and the true label r(c, f (G θ (c, z))) for each dimension of C. Comparison of CZ-GEM and CGAN on face dataset is shown in Figure 9. CGAN not only produces blurry faces but also shows more undesired identity changes. In order to show the shape variation clearly, we provide a zoomed-in view in Figure 10. We provide additional for supervised and unsupervised on the chair dataset from in Figure 11 and Figure 12 respectively. The observation is the same with the previous one. CZ-GEM varies the control variables without changing the shape of chairs. In the first row in Figure 11, the leg of the chairs are visually indistinguishable showing an excellent disentanglement between C and Z. For the in unsupervised setting showing in Figure 12, CZ-GEM is able to disentangle the rotation of chairs without any label. Additional of latent traversal of CZ-GEM in the unsupervised setting is provided in Figure 13. The model is able capture the rotation but the translation is not very smooth.

Hierarchical generative model (hybrid of VAE and GAN) that learns a disentangled representation of data without compromising the generative quality.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.