Question: Represent the article for finding a single-sentence TLDR

Question: In this paper, we show how novel transfer reinforcement learning techniques can be applied to the complex task of target-driven navigation using the photorealisticAI2THOR simulator. Specifically, we build on the concept of Universal SuccessorFeatures with an A3C agent. We introduce the novel architectural1contribution of a Successor Feature Dependent Policy (SFDP) and adopt the concept of VariationalInformation Bottlenecks to achieve state of the art performance. VUSFA, our final architecture, is a straightforward approach that can be implemented using our open source repository. Our approach is generalizable, showed greater stability in training, and outperformed recent approaches in terms of transfer learning ability. The human's ability of navigating unknown spaces (e.g. a firefighter finding the fire hydrant very quickly) primarily relies on visual perception, as well as on previous experience and heavy training . In robotics, we would like to mimic this human behaviour. The advancement of visual navigation algorithms essentially contribute to the prevalence and mobility in robotics and therefore, many different approaches are being explored. Previous research has studied map-based, map-building, and map-less approaches (; ;). In the past, map-based and map-building approaches have been favoured. However, they heavily depend on an accurate mapping of the environment. Also, it requires a carefully executed human-guided training phase which limits its generalizability . With recent advances in Deep Reinforcement Learning (DRL) (;, map-less navigation has experienced major advancements). It has been demonstrated that DRL-based methods are now able to solve navigation tasks in a more human-like manner . Research has shown that DRL-based navigation, in particular target driven visual navigation, is still a challenging task especially when targets are represented in the form of visual information that is highly dynamic. In previous navigation paradigms, the agent navigates to a target demonstrating specific properties (e.g. a yellow cone, such as in the case of), whose location may change over time. In contrast, in target driven visual navigation, the agent should be able to learn to navigate in a persistent state space to a dynamic set of goals. The agent is required to learn to navigate when both the goal and the current state are presented as visual images. A current challenge for DRL algorithms is learning new tasks or goals that vary from what the agent was initially trained for. This ability is called transfer learning. There are two popular strategies for achieving transfer learning in DRL, either by using the concept of General Value Functions (GVF) or by using Successor Feature Approximation (SFA) . For the task of target driven visual navigation, demonstrated that an A3C agent using the concept of GVF can improve the transfer learning ability. GVF does not however allow us to easily see the underlining process of learning the dynamics of tasks and GVF agents also frequently struggle in complex environments . The second strategy, applying SFA, enables us to capture the dynamics of the environment by attempting to learn future state visitations, although these also encounter limitations when facing multiple tasks. Universal Successor Features Approximators (USFA), which is an extension of SFA, is able to consider multiple tasks and can improve the transfer learning ability of the agent. In summary, our research contribution is threefold: • For the first time in the literature, we apply Universal Successor Feature Approximators (USFA) for the complex task of target driven visual navigation. Our new approach provides a stable training mechanism and enhances the transfer reinforcement learning ability in complex environments. • We introduce the concept of a Successor Feature Dependant Policy (SFDP), a novel architectural contribution in which the policy can directly make use of the information presented by USFA (an abstract map in our case). This important add-on significantly improves the transfer learning ability of the DRL agent. • Finally, we contribute Variational Universal Successor Feature Approximators (VUSFA), by adopting the concept of Variational Information Bottlenecks. We show that this combination works stably with complex tasks such as target driven visual navigation in the photo-realistic AI2THOR environment. Besides stable convergence, our approach shows possible ways in which transfer learning could be improved in the future. Transfer in reinforcement learning agents can be described as the ability of an agent to generalize over different tasks while sharing knowledge between them. In this paper we tackle the problem of transfer reinforcement learning with respect to Successor Feature Reinforcement Learning (SF-RL) . Specifically we develop on the concept of Universal Successor Feature RL (USF-RL) (b) which is an extension of SF-RL. In the next section we will first introduce basic concepts where we have used throughout our research. We can formalize the goal-directed navigation task as a Markov Decision Process (MDP). The transition probability p(s t+1 |s t, a t) defines the probability of reaching the next state s t+1 when action a t ∈ A is taken in state s t ∈ S. For any goal g ∈ G (in our case G ⊆ S), we define a goal dependent reward function r g (s t, a t, s t+1) ∈ R and a discount function γ g (s t) ∈ (for terminal state, γ g = 0). For any policy π(a t |s t), a GVF can be defined as follows: The assumption for any goal g is that there exists an optimal value function V π * g g (s), which is evaluated according to a goal oriented optimal policy π * g. The general aim of agent's learning is to find the optimal policy π * that maximises the future discounted rewards starting from s 0 and following π *. To generalize over the goal space G, the agent needs to learn multiple optimal policies as well as optimal value functions in order to navigate to a goal. Each goal is considered a new task and the agent should be able to quickly adapt to find V π * g g (s) and π * g. Universal Successor Features (USF) (b) in an extension of the idea of Successor Features (SF) described in and. Similar to the concept of SF, USF also follows the idea that the immediate scalar reward r g can be defined as a linear combination of state representations φ and a goal dependent reward prediction vector ω g as in Equation 2. In the Equation 2, φ (s t, a t, s t+1) represents the dynamics or the physical features the agent sees when transitioning between states s t and s t+1 after taking an action a t. We approximate φ (s t, a t, s t+1) as φ (s t+1) following Ma et al. (2018a); since it is convenient for the agent to rely on the state representation of the new state φ (s t+1) to recover the scalar reward r g rather than trying to capture physical features of transition dynamics. This allows us to describe the value function as a cumulative sum of the discounted φ as follows: where ψ π g (s t) is defined as the Universal Successor Features (USF) of state s t (b). Intuitively, ψ π g (s t) can be thought of as the expected future state occupancy. Unlike traditional Successor Feature Approximation, USF is based on both the state and the goal. The value function defined with USFA has similar properties to GVFs. The modified V π g (s) with ψ incorporates shared dynamics between tasks. Learning the USFA is accomplished in the same way as the value function update by using the following TD (Temporal Difference) error: As illustrated by (b) the vectors ψ π g (s t), φ (s t+1) and ω g can be approximated by neural networks parameterized by θ π, θ φ and θ ω. In this paper we incorporate the concept of USF with an A3C agent and trained all three sets of parameters jointly. The USFA model introduced by Ma et al. (2018b) extends the concept of SF generalized over multiple tasks with the actor-critic algorithm. However, their method is yet to be evaluated with complex tasks such as target driven visual navigation. In this section we point out the challenges when adapting Ma et al.'s model to complex tasks. The state representation φ vector mentioned in the USF architecture plays a crucial role. It decouples the scalar reward r t and learns the USFA ψ π g (s t) with the TD-error loss function (Equation 4). The authors Ma et al. (2018b) propose learning φ using an autoencoder prior to the main reinforcement learning algorithm. φ is supposed to capture the salient information about each state s t, but when the states consist of complex visual information such as photo-realistic images, defining an optimal φ representation with an autoencoder can be problematic. Training a convolutional autoencoder which generalize over many states is often prone to over-fitting. Training ω g by a regression loss with respect to a scalar reward r t (Equation 2) can also be problematic. The main reason is that this loss is not informative enough to train ω g because during the initial stages of training, the agent will observe very small negative rewards and rarely see the large positive reward goal locations. ω g when decoupled from the scalar reward, captures information about the goal. Ma et al. (2018b) propose to train ω g with a separate neural network that uses goal features as input. Training a separate network in our scenario easily leads to over-fitting on the limited number of trained goal locations seen by the agent during training and leads to poor generalization of the agent to new unknown goals. Our first contribution is the application of USFA for the complex task of target driven visual navigation. This section introduces how we created a stable architecture while addressing the aforementioned issues. Rather than using a separate network such as an autoencoder to generate φ, we argue it is more beneficial if the agent learns task dependant state representation features while exploring the state space. Thereby, the agent should learn to capture only the salient features relevant to the task of navigation and ignore features that may be solely important for reconstruction. Since in target driven visual navigation, the goal space is a subset of the state space, we used a siamese network to generation both φ and ω g. A major problem with training a stable USFA based DRL agent is the difficulty of training an ω g that works successfully with the scalar reward regression loss (see Equation 2). In our case, the reward structure is ad-hoc: for every time-step the agent either receives a small negative penalty or a large reward for reaching the goal location. The positive rewards are experienced by the agent much less frequently, particularly at the beginning of training. When training large scale tasks, this class imbalance can be even more detrimental because the reward structure needs to be learnt before the reinforcement learning agent is able to learn a meaningful policy for navigation. If not, this creates an unstable ω g which can cause the network to diverge. To overcome this problem, we propose to exploit the A3C agent's critic update (Value function update). In a conventional A3C algorithm, each agent needs to optimise the critic and policy functions after each episode with the N-step Return . Since the value function can be interpreted by the USFA concept as being a linear combination of ψ g (s t) and ω g, the critic's loss function in an A3C agent can be used to learn ω g. Unlike training the network with a scalar reward regression loss, this method is more informative because the loss function depends on the episode's discounted scalar rewards. The discounted return calculation for a single episodic step in A3C is depicted in Algorithm 1 in the Supplementary Materials. Equation 5 shows how the value function can be decoupled with ψ and ω. Equation 6 shows the conventional one step TD loss for the SFA branch. It needs to be highlighted that ψ gets updated with both Loss ψ T D and Loss V T D. To counter the problem of the having only a few training goals to train ω, we utilised the embeddings generated from the Siamese layer as goal information and trained ω as another branch of the USFA-A3C agent (see Figure 5 in the Supplimentary Materials). Our second contribution is the addition of a Successor Feature Dependant Policy (SFDP) to the USFA implementation. As mentioned before, ψ g (s t) can be seen as an abstract representation of the cumulative sum of the future states the agent will visit by following an optimal policy . Traditionally, successor features are not directly consulted when determining an action (b). However, we hypothesise that feeding the abstract map of future states could be useful in determining the next action. USF can be described as representing the cumulutive sum of discounted future states the agent visits following an optimal policy. This property by itself helps with transfer learning because eventhough different goals have different optimal paths, they can share some common sub-paths. For example, when tasked with finding the microwave and sink in a kitchen, the initial steps of the agent in going to the kitchen will be similar for both tasks. We hypothesised that if the policy has direct access to the USF (see Equation 7), the agent will be able to learn from these similar paths. By directly concatenating ψ g with the final layer of the policy head naively in ψ g being updated with gradients from the conventional bellman optimality Equation 3 and the policy gradients Figure 1: Proposed Network Architecture "VUSFA": The model's input is the current state of the agent s t and the goal location g as images. These go through a shared simaese encoder E(z|s t). The reparametrized output z is used to train the ω vector. The policy is conditioned on the USF vector (dotted line indicates gradients do not flow from policy to the USFA head). The USFA ψ is trained with the temporal difference error using φ to give the expected future state occupancies. The discounted episode return is used to train both ω and USFA vectors. of the A3C agent. This can harm the true USF representation and can reduce the transfer learning capabilities of the agent. Therefore in the final model, we stopped the gradient flow from the policy head to the USF branch. The stopping of policy gradients for the USF branch is illustrated in Figure 1 with dotted lines. The next modification we made was the introduction of the Variational Siamese Bottleneck (VSB) to improve the quality of USFA and the reward prediction vector. We observed that the embeddings generated by the siamese layers play a key role in improving the overall performance due to their effect on generating ψ and ω g. We wanted to improve these embeddings without harming stable convergence. Our main hypothesis in selecting the variational information bottleneck was that it will be able to guide the Siamese layers to extract the most informative and meaningful features which will then lead to better generalisation. Having a siamese layer that generates embeddings which produce a robust φ and ω g is key to improving the transfer learning ability of the overall model. If these embeddings are not informative enough, the model can overfit to the training set of goals. To improve the embeddings without harming the training stability, we adapt the concept of Deep Variational Information Bottleneck . Our show that this addition improves the performance of the overall network. In the next sections we will describe the theory behind the Variational Information Bottleneck and the training procedure we used in our adaptation of it to the VUSFA agent. The theory of the Information Bottleneck, introduced by , shows that a deep neural network can be thought of as a trade-off between having a compressed latent representation Z with respect to inputs X while still preserving relevant information with respect to the outputs Y. Mathematically, this idea of generating an optimal Z can be achieved using Mutual Information Theorem as follows: Where I(X; Z) is the mutual information between the input features X and the latent representation Z from the hidden layer and I(Y ; Z) is the mutual information between output Y and Z. Intuitively, the neural network should predict the output while reducing the mutual information between input and the encoding. The minimisation of I(X; Z) encourages the agent to compress the most relevant information about X into Z for the prediction of Y. Deep Variational Information Bottlenecks introduced by is a parameterized approach to the Information Bottleneck theory that can be easily used with deep neural networks. This was done by introducing a regularized objective function as follows. Minimizing the loss function J(q(Y |Z), E(Z|X)) encourages the neural network to generate an informative compressed embedding Z from the input X. Equation 9 consists of a parametric encoder function E(Z|X) that maps input features X into latent vector Z, a decoder function q(Y |Z) that maps Z to output labels Y, and a mutual information constraint I(X, Z) ≤ I c. The generation of Z by the encoder under the Information Constraint I c can be though of as a bottleneck layer in the network. This bottleneck Z could be applied as any intermediate layer of the neural network. In our scenario, we applied the Information Bottleneck on the output of the siamese layers (see Figure 1) due to it's direct effects on the estimations of π, ψ, and ω. The siamese layers can therefore be thought of as the encoder E(Z|X). We call this the Variational Siamese Bottleneck (VSB). The VSB enforces an upper-bound I c on the mutual information term I(Z, X) to encourage the encoder E(Z|X) to focus on the most discriminative features of the input. The encoder needs to generate a latent distribution Z where the mutual information between X and Z does not exceed the scalar Information Constraint I c where I c is a hyperparamter. Since the I(Z, X; θ) ≤ I c term in Equation 9 is intractable, we cannot directly apply it to a neural network trained with back-propagation. introduced a modified version by applying a variational lower bound and a lagrangian multiplier β that needs to be updated adaptively. This in the final deep variational information bottleneck loss in Equation 10. Since the KL divergence is calculated between two distributions, the encoder outputs the mean and variance of Z from which a sample is taken using the reparametrization trick. We update the Lagrangian multiplier β in a similar way to. β gets updated for each actor thread adaptively following Equation 11. The final loss function of our agent with the Variational Information Bottleneck is shown in the Equation 12. -where L total is the combined loss function Therefore, the agent needs to minimize both L total and the KL divergence term [KL[E(z|x) r(z)] − I c ] at the same time. λ π,λ ψ and λ V are hyperparameters. Our network (see Figure 1) takes the four most recent states the agent has visited as s t and four repeated goal states as the g. Then the resnet-50 embeddings related to both s t and g go through a siamese encoder and generates the mean and variance vectors, which are the parameters of the latent distribution Z. The φ embeddings with respect to the goal g then go through a fully connected layer to predict the goal coefficients ω vector. The network's last part predicts the policy π(a t |s t, g, ψ) and the USFA ψ g (s t). Similar to, we include a separate policy (π), USFA ψ and the expected sum of future rewards prediction heads for each scene (e.g. Bathroom). 6.1 TRAINING VUSFA The training procedure for our model is based on the A3C algorithm and is shown in Algorithm 1. The reparameterized embedding was not directly used in predicting the policy π and the USFA ψ to maintain a stable procedure. Instead, the mean vectors of the state representation φ for both goal and state were used. These mean vectors were concatenated together and fed through the layers used for predicting the policy and USFA as shown in Figure 1. We used the reparameterized embeddings from the bottleneck layer to predict ω since the ω vector is the most important element in the USF architecture that decouples the value function. The objective behind this reparametrization was to make create an ω that is robust and generalizable that would not easily overfit. During inference, we use reparameterized values for both goal and state encoding which we assume added more generalizability and exploration and improved zero-shot navigation of the agent. The evaluation of the agent under the task of target driven visual navigation has been conducted in two ways. First, the agent was evaluated on its zero-shot learning ability. The second evaluation criteria was the time taken for the agent to adapt to new unknown goals when fine-tuning. Both evaluation criteria belong to the domain of Transfer in Reinforcement Learning and will be described in the following two sections. Prior to evaluation, all models were trained on four scenes for 20 different goals until convergence. We took the deep Siamese A3C model by as the baseline, since it is the most relevant work done using the AI2THOR simulator. Moreover, we were also successful in training the agent from scratch with a CNN (replacing the resnet features). Although adding an LSTM in further performance increases, we used resnet features instead. We decided to do so to keep our training time low and the evaluation consistent. We evaluated all variations of our proposed model in a benchmark with the state-of-the-art: Model 04: Adding VSB to Model 03 (we call this VUSFA) The aim of zero-shot navigation is to see weather the agent is be able to reach a wide range of goals while being trained on only a very limited subset of goals. In particular, the zero-shot learning capability of the agent was evaluated by testing the agent's average successful attempts to find new goal locations. In the evaluation process, we follow a similar criteria to, in which we tested whether the agent was able to reach the goal in less than 500 time-steps. We constituted this as a successful episode. We evaluated the success rate of reaching all goals in each environment. We repeated this procedure 10 times (trials), in which the agent always started from a random location. We trained our models on only 20 goal states spread evenly across four environments using the AI2THOR simulator. This represents less than 1.2% of the total number of states. In-spite of this, even the worst performing model was able to generalize to over 16% of all states. Table 1 shows that all proposed algorithms (Model 02-04) are able to successfully reach more locations without training than the baseline Model 01. The USFA-based policies consistently generalise better than. It can take a large amount of time (in the order of several days), to train a complex DRL agent. Therefore, it is impractical to re-train agents when the task slightly changes. Instead, the agent should be able to use previous knowledge to adapt to new tasks quickly. We evaluated the transfer learning ability of all four models to 20 new goals. In order to evaluate how the closeness of the new goals effect the agent's performance, we tested the models on states that are 1, 2, and 4 steps away from already trained goals as well as completely random goals. We sampled 5 random states from each environment, excluding the already trained goals to get the new goals. We used 5 trials, meaning repeating this process 5 times with random states which are different to the previously learned ones. To ensure a fair comparison, we kept the random seeds constant between the models. Figure 2 shows the number of time-steps required for the model to adapt to new goals. It becomes clear that the USFA-based policies are consistently able to decrease the number of steps taken to reach the goal faster than the baseline model. Moreover, using the SFDP with USFA ed in a further decrease in time-steps required and thus showed to have a positive effect on the model's transfer learning ability. As shown in Figure 2, VUSFA is usually able to further improve performance. We proposed Variational Universal Successor Features Approximator (VUSFA) to solve rather complex tasks, such as target driven visual navigation in photorealistic environments using the AI2THOR simulator. To our knowledge, this is the first time the Deep Variational Information Bottleneck theory has been applied with Universal Successor Features in Deep Reinforcement Learning. Our indicate that VUSFA is able to improve the transfer learning ability in respect to previous state-of-the-art GVF and USF-RL based research. Our approach is generalizable and can be easily adapted to various tasks other than navigation. For re-implementation, we provide the source code via our github repository 1. Our approach introduces a new perspective and should be considered in future research aiming to improve transfer learning for Deep Reinforcement Learning. In particular, further research could look into exploration of the semantical impacts of φ, ω, and ψ. Algorithm 1 Variational A3C-USF Algorithm pseudocode for each actor thread Assuming global parameter vectors for value function and policy as θ π, θ ψ and θ ω Assuming global shared counter as T = 0 Assuming thread specific parameter vectors for value function and policy as θ π, θ ψ and θ ω d Initialize thread step counter t ← 1 1: repeat 2: Reset gradients: Synchronise thread specific parameters with global network θ π = θ π, θ ψ = θ ψ, θ ω = θ ω 4: Get initial state s t, goal g 6: Perform an action a t according to the current policy π(a t |s t, g : θ π) Receive the scalar reward r t,the new state s t+1 Collect roll-outs [s t, r t, s t+1] t ← t + 1 11: until terminal s t or t − t start == t max Bootstrapping the return R from last state of the episode 14: for i ∈ t − 1,...,t start do 17: Computing dθ ψ,dθ π and dθ ω, perform asynchronous update of θ ψ,θ π and θ ω 28: until T > T max We used the fundamental A3C framework to train our agent. The initial A3C agent contains 100 threads assigned to 20 goals (5 goals from each scene). The 20 goals used for training can be seen in Figure 3. For the transfer learning tasks, 20 new goals were randomly selected excluding the trained goals. An ideal trajectory of an agent is illustrated in the Figure 4. For training, we used a high performance computer with 4x Xeon 6136 @3Ghz Processor (total of 48 cores); Memory -1.18TB, GPU -Tesla V100 with 32GB memory per GPU. We did limited hyperparameter tuning as detailed in Table 2. λ V = 0.5, λ ψ = 0.0005 and I c = 0.2 gave good and were used in the plots. Values Explored Values Used Figure 5: Architecture used for the SFDP-A3C model The input to the model is the current state of the agent s t and the goal location g as images. These go through a shared siamese layer to generate embeddings for the goal and the state. The policy is conditioned on the USF vector (dotted line indicates gradients do not flow from policy to the USFA head). The USFA ψ is trained with the temporal difference error using φ to give the expected future state occupancies. The discounted episode return is used to train both ω and USFA vector. Designing a value function that is capable of adapting to different tasks is based on a concept known as General Value Functions (GVF) . An extension to this idea, called Universal Value Function Approximators (UVFAs), was introduced by. The core idea of UVFAs is to represent a large set of optimal value functions by a single, unified function approximator that generalises over both states and goals. Although theoretically sound, learning a sufficiently working UVFA is a challenging task (b). Successor Representation (SR) emerged from the field of cognitive science and is modelled on the brain's capability to create a reusable predictive map. Recently, SR was combined with Deep Learning to create Deep Successor Reinforcement Learning by. Based on Deep Q-Network (DQN) fundamentals, Kulkarni et al. (2016 were able to learn task-specific features that were able to quickly adapt to distal changes in the reward by fine-tuning only the reward prediction feature vector. Transfer in RL has been evaluated on multiple similar tasks by , who introduced Successor Features (SF). They adapted SR to be applicable in the continuous domain and were able to show how classic policy improvement can be extended to multiple policies. to Deep Learning, additionally showed how these models can be stably trained. For the problem of visual navigation, a SR-based DRL architecture similar to was used by. Unlike our approach, they showcase their solution in simple maze-like environments using DQN as the baseline method, while we use actor-critic methods in a photorealistic simulation environment. DQN-based techniques frequently suffer from stability issues when applied to complex problems, such as large-scale navigation. USFA (b) is the most recent extension to SR. Unlike previous methods, which were based on DQN, USFA learns a policy directly by modeling it with actor-critic methods. Similar to SR, USFA modifies the policy with successor features. DQN-based approaches learn an optimal action-value function indirectly. USFA attempts to obtain a GVF which can be directly used to obtain an optimal policy. It can be seen as a combination of the SF and UVFA methods as discussed earlier. Unlike methods based on DQN, USFA is able to directly optimize an agent to learn multiple tasks simultaneously. However, USFA has not been tested on high-dimensional complex problems. This paper shows the adaption of USFA to more complex task for target driven visual navigation in a photorealistic environment. also extended the idea of SF-RL in to USFA by adapting the concepts of GVF and generalized policy improvement theorem. work describes a USFA training method by adapting ε-greedy Q learning for set of simple tasks in Deepmind-Lab simulator. We found their method is hard to adapt to our problem domain mainly due to defining the state representation vector as a linear combination of a set of tasks. This method can work with a simple set of known tasks such as collecting different objects in the simulator. But in our case, each new task is similar to a new goal location selected from the state space. Also, the authors have not extended this algorithm to work with policy gradient methods. Similar to the UVFA , a goal-dependant SF is approximated with Universal Successor Feature Approximators (USFA) (b; . The goal dependant USF obeys the bellman update rule similar to SF (Equation 13).. In Equation 14 ψ π (s t, g t ; θ ψ), ω(g t ; θ ω) and φ (s t, a t, s t+1 ; θ φ) should generalize over both goals and states. The fundamental architecture to learn USF was proposed by Ma et al. (2018b) and is shown in Figure 6. The architecture consists of three sub-networks that need to be trained. The first one is the state representation network with parameters Φ. This network is an autoencoder which gets trained before the training of the DRL agent. Then there are two heads to approximate the USF (θ π), and policy (θ ψ). These two heads share parameters in the early layers with a fork in the last fully connected layer. Finally, θ w is learned. This final reward prediction network should be able to generate ω for different goals which will later be used to train the USF agent. It is important to compare the architectural differences between the USF-RL methods (b) with the previous DQN based SF-RL methods . Unlike in SF-RL, φ is not learned simultaneously while training the DRL agent. The φ prediction network in the USF-RL agent is trained with an autoencoder separately prior to the training phase. In contrast to the SF-RL methods which stores an ω vector for each task, in USF-RL the ω vector is predicted with a network that takes information about the goal as the input. Maintaining a separate network to generate ω given goal information gives the luxury for the agent to quickly adapt to novel goals. In Equation 15 the term I(X, Z) is the mutual information between the features X and the hidden layer Z. I(Y, Z) represents the mutual information between output Y and the hidden layer Z. Intuitively, Equation 15 says that the neural network should predict the label while reducing the mutual information between the input and the encoding. Reducing of the I(X, Z) term encourages the agent to compress the X into Z where Z consists of only the most important information about the input features. The β is the trade-off hyperparameter which controls the quality of Z. The mutual information term I(X, Z) between input features and the encoder embeddings can be illustrated in Equation 16. I(X, Z) = p(x, z) log(p(x, z) p(x)p(z) )dxdz = p(x)E(z|x) log(E(z|x) p(z) )dxdz In Equation 16, the joint distribution of p(x, z) can be illustrated with the encoder E(z|x) and the input distribution p(x). Another important factor in Equation 16 is the distribution of the latent variable p(z) is intractable. So the p(z) term is replaced with a known prior distribution r(z). This assumption introduces an upper-bound to the mutual information term I(X, Z) as mentioned in Equation 17. I(X, Z) ≤ p(x)E(z|x) log(E(z|x) r(z) )dxdz In Equation 17, the sum of the probability distribution of input X can be transformed into an expectation operation. The expectation operation simplifies the I(X, Z) into a KL divergence between the distribution of Z generated by the parametric encoder and the approximated prior r(z). Modification of the I(X, Z) with the KL divergence makes it easier to train a neural network with the loss function mentioned in Equation 9. The interpretation of I(X, Z) with KL divergence allow us to modify the conditional loss function β that needed to be updated in an adaptive manner with the training procedure of the neural network. Moreover, Alemi et al. evaluated the method of supervised learning tasks and showed that the models trained with a VIB could be less prone to be overfitting and more robust to adversarial examples. In this paper, we successfully illustrated the adoption of the concept of Deep VIB with the USF to substantially improve the transfer learning ability of a navigation agent.

Represent the short summary for finding the corresponding scientific article (long)

We present an improved version of Universal Successor Features based DRL method which can improve the transfer learning of agents.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.


Question: Represent the article for finding a single-sentence TLDR

Question: A major challenge in learning image representations is the disentangling of the factors of variation underlying the image formation. This is typically achieved with an autoencoder architecture where a subset of the latent variables is constrained to correspond to specific factors, and the rest of them are considered nuisance variables. This approach has an important drawback: as the dimension of the nuisance variables is increased, image reconstruction is improved, but the decoder has the flexibility to ignore the specified factors, thus losing the ability to condition the output on them. In this work, we propose to overcome this trade-off by progressively growing the dimension of the latent code, while constraining the Jacobian of the output image with respect to the disentangled variables to remain the same. As a , the obtained models are effective at both disentangling and reconstruction. We demonstrate the applicability of this method in both unsupervised and supervised scenarios for learning disentangled representations. In a facial attribute manipulation task, we obtain high quality image generation while smoothly controlling dozens of attributes with a single model. This is an order of magnitude more disentangled factors than state-of-the-art methods, while obtaining visually similar or superior , and avoiding adversarial training. A desired characteristic of deep generative models is the ability to output realistic images while controlling one or more of the factors of variation underlying the image formation. Moreover, when each unit in the model's internal image representation is sensitive to each of these factors, the model is said to obtain disentangled representations. Learning such models has been approached in the past by training autoencoders where the latent variables (or a subset of them) are constrained to correspond to given factors of variation, which can be specified (supervised) or learned from the data (unsupervised) BID22 BID29 BID15. The remaining latent variables are typically considered nuisance variables and are used by the autoencoder to complete the reconstruction of the image. There exists one fundamental problem when learning disentangled representations using autoencoders, sometimes referred to as the "shortcut problem" BID29. If the dimension of the latent code is too large, the decoder ignores the latent variables associated to the specified factors of variation, and achieves the reconstruction by using the capacity available in the nuisance variables. On the other hand, if the dimension of the latent code is small, the decoder is encouraged to use the specified variables, but is also limited in the amount of information it can use for reconstruction, so the reconstructed image is more distorted with respect to the autoencoder's input. BID29 showed that this trade-off between reconstruction and disentangling can indeed be traversed by varying the dimension of the latent code. However, no principled method exists to choose the optimal latent code dimension. The shortcut problem was also addressed by using additional mechanisms to make sure the decoder output is a function of the specified factors in the latent code. One approach, for example, consists in swapping the specified part of the latent code between different samples, and using adversarial training to make sure the output distribution is indeed conditioned to the specified factors BID22 BID19 BID29. However, adversarial training remains a difficult and unstable optimization problem in practice. Based on these observations, we propose a method for avoiding the shortcut problem that requires no adversarial training and achieves good disentanglement and reconstruction at the same time. Our method consists in first training an autoencoder model, the teacher, where the dimension of the latent code is small, so that the autoencoder is able to effectively disentangle the factors of variation and condition its output on them. These factors can be specified in a supervised manner or learned from the data in an unsupervised way, as we shall demonstrate. After the teacher model is trained, we construct a student model that has a larger latent code dimension for the nuisance variables. For the student, we optimize the reconstruction loss as well as an additional loss function that constrains the variation of the output with respect to the specified latent variables to be the same as the teacher's. In what follows, we consider autoencoder models (E, D), that receive an image x as input and produce a reconstructionx: D(E(x)) =x. We consider that the latent code is always split into a specified factors part y ∈ R k and a nuisance variables part z ∈ R d: E(x) = (y, z), D (y, z) =x. Consider a teacher autoencoder (E T, D T), with nuisance variables dimension d T, and a student DISPLAYFORM0 Because the dimension of the nuisance variables of the student is larger than in the teacher model, we expect a better reconstruction from it (i.e. ||x −x S || < ||x −x T ||, for some norm).At the same time, we want the student model to maintain the same disentangling ability as the teacher as well as the conditioning of the output on the specified factors. A first order approximation of this desired goal can be expressed as DISPLAYFORM1 where j ∈ {1...H · W · C}, H, W and C are the dimensions of the output image, and i ∈ {1...k} indexes over the specified factors of variation. In this paper we propose a method to impose the first-order constraint in, which we term Jacobian supervision. We show two applications of this method. First, we propose an unsupervised algorithm that progressively disentangles the principal factors of variation in a dataset of images. Second, we use the Jacobian supervision to train an autoencoder model for images of faces, in which the factors of variation to be controlled are facial attributes. Our ing model outperforms the state-of-theart in terms of both reconstruction quality and facial attribute manipulation ability. Autoencoders BID12 are trained to reconstruct an input image while learning an internal low-dimensional representation of the input. Ideally, this representation should be disentangled, in the sense that each hidden unit in the latent code should encode one factor of variation in the formation of the input images, and should control this factor in the output images. There exist extensive literature on learning disentangled representations BID27 BID1 BID5 BID7 BID4 BID22 BID25 BID15 BID3.Disentangled representations have two important applications. One is their use as rich features for downstream tasks such as classification BID27 BID30 or semi-supervised learning. In the face recognition community, for example, disentanglement is often used to learn viewpoint-or pose-invariant features BID32 BID24 BID30. A second important application is in a generative setting, where a disentangled representation can be used to control the factors of variation in the generated image BID25 BID11 BID22 BID19. In this work we concentrate on the second one. In recent years, with the advent of Generative Adversarial Networks (GANs) BID9, a broad family of methods uses adversarial training to learn disentangled representations BID22 BID19 BID25 BID4. In a generative setting, the adversarial discriminator can be used to assess the quality of a reconstructed image for which the conditioning factors do not exist in the training set BID22 BID4.Another alternative, proposed in Fader Networks BID19, is to apply the adversarial discriminator on the latent code itself, to prevent it from containing any information pertaining to the specified factors of variation. Then, the known factors of variation or attributes are appended to the latent code. This allows to specify directly the amount of variation for each factor, generating visually pleasing attribute manipulations. Despite being trained on binary attribute labels, Fader Networks generalize remarkably well to real-valued attribute conditioning. However, despite recent advances BID10, adversarial training remains a non-trivial min-max optimization problem, that in this work we wish to avoid. Other remarkable disentangling methods that require no adversarial training are: BID5, where the cross-covariance between parts of the latent representation is minimized, so that the hidden factors of variation can be learned unsupervised and BID11; BID15; BID3 where a factorized latent representation is learned using the Variational Autoencoder (VAE) framework. In particular, the authors of BID3, propose to overcome the disentangling versus reconstruction trade-off by progressively allowing a larger divergence between the factorized prior distribution and the latent posterior in a VAE.Related to the task of varying the factors of image generation is that of domain-transfer BID6 BID8 BID20. Here the challenge is to "translate" an image into a domain for which examples of the original image are unknown and not available during training. For example, in the face generation task, the target domain can represent a change of facial attribute such as wearing eyeglasses or not, gender, age, etc. BID20 BID25 BID6. In this section we detail how the Jacobian supervision motivated in Section 1 can be applied, by ways of a practical example. We will use the Jacobian supervision to learn a disentangled image representation, where the main factors of variation are progressively discovered and learned unsupervised. We start with a simple autoencoder model, the teacher T, identified by its encoder and decoder parts (E T, D T). The output of the encoder (the latent code) is split into two parts. One part corresponds to the factors of variation y ∈ R k and the other part corresponds to the nuisance variables, z ∈ R d.We begin by using k = 2 and d = 0, meaning that the latent code of the teacher is only 2-dimensional. We consider the information encoded in these two variables as the two principal factors of variation in the dataset. This choice was done merely for visualization purposes FIG0 ).For this example, we trained a 3-layer multi-layer perceptron (MLP) on MNIST digits, using only the L2 reconstruction loss. We used BatchNorm at the end of the encoder, so that the distribution of y is normalized inside a mini-batch. In FIG0 (a) we show the of sampling this twodimensional variable and feeding the samples to the decoder D T. The ing digits are blurry, but the hidden variables learned to encode the digit class. Next, we create a student autoencoder model (E S, D S), similar to the teacher, but with a larger latent code. Namely, k = 2 and d = 1 instead of d = 0, so that the latent code has now an extra dimension and the reconstruction can be improved. In order to try to maintain the conditioning of the digit class by the 2D hidden variable y, we will impose that the Jacobian of the student with respect to y be the same as that of the teacher, as in. How to achieve this is described next. We take two random samples from the training set x 1 and x 2, and feed them to the student autoencoder, producing two sets of latent codes: (y the same pair of images to the teacher autoencoder to obtain y DISPLAYFORM0 Note that the teacher encoder in this case does not produce a z. We observe, by a first-order Taylor expansion, that DISPLAYFORM1 and DISPLAYFORM2 where J T and J S are the Jacobian of the teacher and student decoders respectively. Suppose y DISPLAYFORM3 and DISPLAYFORM4 then, by simple arithmetic, DISPLAYFORM5 where, since we assume holds, we dropped the superscripts for clarity. What FORMULA7 expresses is that the partial derivative of the output with respect to the latent variables y in the direction of (y 2 − y 1) is approximately the same for the student model and the teacher model. To achieve this, the proposed method consists essentially in enforcing the assumptions in FORMULA5 and FORMULA6 by simple reconstruction losses used during training of the student. Note that one could exhaustively explore partial derivatives in all the canonical directions of the space. In our case however, by visiting random pairs during training, we impose the constraint in for random directions sampled from the data itself. This allows for more efficient training than exhaustive exploration. Putting everything together, the loss function for training the student autoencoder with Jacobian supervision is composed of a reconstruction part L rec and a Jacobian part L jac: DISPLAYFORM6 Figure 2: 3 rd to 6 th principal factors of variation discovered by our unsupervised algorithm. The first two factors of variation are learned by the first teacher model FIG0 ). Each time a hidden unit is added to the autoencoder, a new factor of variation is discovered and learned. Each row shows the variation of the newly discovered factor for three different validation samples, while fixing all the other variables. The unsupervised discovered factors are related to stroke and handwriting style.where the subscript j indicates a paired random sample. For the experiments in FIG0 we used λ y = 0.25, λ dif f = 0.1. TAB4 in the appendix presents ablation studies on these hyperparameters. In practice, we found it also helps to add a term computing the cross-covariance between y and z, to obtain further decorrelation between disentangled features BID5: DISPLAYFORM7 where M is the number of samples in the data batch, m is an index over samples and i, j index feature dimensions, andz i andȳ j denote means over samples. In our experiments we weigh this loss with λ xcov = 1e −3.Once the student model is trained, it generates a better reconstructed image than the teacher model, thanks to the expanded latent code, while maintaining the conditioning of the output that the teacher had. The extra variable in the student latent code will be exploited by the autoencoder to learn the next important factor of variation in the dataset. Examples of factors of variations progressively learned in this way are shown in Figure 2.To progressively obtain an unsupervised disentangled representation we do the following procedure. After training of the student with k = 2, d = 1 is finished, we consider this model as a new teacher (equivalent to k = 3), and we create a new student model with one more hidden unit (equivalent to k = 3, d = 1). We then repeat the same procedure. Results of repeating this procedure 14 times, using 100 epochs for each stage are shown in FIG0. In FIG0 (b), we show how the ing final model can maintain the conditioning of the digit class, while obtaining a much better reconstruction. A model trained progressively until reaching the same latent code dimension but without Jacobian supervision, and only the cross-covariance loss for disentangling BID5, is shown in FIG0 (c). This model also obtains good reconstruction but loses the conditioning. For this model we also found λ xcov = 1e −3 to give the best . To quantitatively evaluate the disentangling performance of each model, we look at how the first two latent units (k = 2) control the digit class in each model. We take two images of different digits from the test set, feed them to the encoder, swap their corresponding y subvector and feed the fabricated latent codes to the decoder. We then run a pre-trained MNIST classifier in the generated image to see if the class was correctly swapped. The quantitative are shown in TAB0. We observe that the reconstruction-disentanglement trade-off is indeed more advantageous for the student with Jacobian supervision. To complement this section, we present of the unsupervised progressive learning of disentangled representations for the SVHN dataset BID23 in Section A.5 in the Appendix. In photographs of human faces, many factors of variation affect the image formation, such as subject identity, pose, illumination, viewpoint, etc., or even more subtle ones such as gender, age, expression. Modern facial manipulation algorithms allow the user to control these factors in the generative process. Our goal here is to obtain a model that has good control of these factors and produces faithful image reconstruction at the same time. We shall do so using the Jacobian supervision introduced DISPLAYFORM0 Figure 3: Diagram of the proposed training procedure for facial attributes disentangling. E and D always denote the same encoder and decoder module, respectively. Images x 1 and x 2 are randomly sampled and do not need to share any attribute or class. Their ground truth attribute labels areȳ 1 and y 2 respectively. The latent code is split into a vector predicting the attributes y and an unspecified part z. Shaded E indicates its weights are frozen, i.e., any loss over the indicated output does not affect its weights.in Section 3. In this more challenging case, the disentangling will be first learned by a teacher autoencoder using available annotations and an original training procedure. After a teacher is trained to correctly disentangle and control said attributes, a student model will be trained to improve the visual quality of the reconstruction, while maintaining the attribute manipulation ability. We begin by training a teacher model for effective disentangling at the cost of low quality reconstruction. Figure 3 shows a diagram of the training architecture for the teacher model. Let x ∈ R H×W ×3be an image with annotated ground truth binary attributesȳ ∈ {−1, 1} k, where k is the number of attributes for which annotations are available. Our goal is to learn the parameters of the encoder (Figure 3, top). Ideally, y ∈ R k should encode the specified attributes of x, while z ∈ R d should encode the remaining information necessary for reconstruction. DISPLAYFORM0 The training of the teacher is divided into two steps. First, the autoencoder reconstructs the input x, while at the same time predicting in y the ground truth labels for the attributesȳ. Second, the attributes part of the latent code y is swapped with that of another training sample (Figure 3, bottom). The randomly fabricated latent code is fed into the decoder to produce a new image. Typically, this combination of factors and nuisance variables is not represented in the training set, so evaluating the reconstruction is not possible. Instead, we use the same encoder to assess the new image: If the disentangling is achieved, the part of the latent code that is not related to the attributes should be the same for the existing and fabricated images, and the predicted factors should match those of the sample from which they were copied. In what follows, we describe step by step the loss function used for training, which consists of the sum of multiple loss terms. Note that, contrary to relevant recent methods BID22 BID19, the proposed method does not require adversarial training. Reconstruction loss. The first task of the autoencoder is to reconstruct the input image. The first term of the loss is given by the L2 reconstruction loss, as in.Prediction loss. In order to encourage y to encode the original attributes of x indicated in the ground truth labelȳ, we add the following penalty based on the hinge loss with margin 1: DISPLAYFORM1 where the subscript [i] indicates the i th attribute. Compared to recent related methods BID25 BID19, the decoder sees the real-valued predicted attributes instead of an inserted vector of binary attribute labels. This allows the decoder to naturally learn from continuous attribute variables, leaving a degree of freedom to encode subtle variations of the attributes. Cycle-consistency loss. Recall our goal is to control variations of the attributes in the generated image, with the ability to generalize to combinations of content and attributes that are not present in the training set. Suppose we have two randomly sampled images x 1 and x 2 as in Figure 3. After obtaining (y 1, z 1) = E(x 1) and (y 2, z 2) = E(x 2), we form the new artificial latent code (y 2, z 1). Ideally, using this code, the decoder should produce an image with the attributes of x 2 and the content of x 1. Such an image typically does not exist in the training set, so using a reconstruction loss is not possible. Instead, we resort to a cycle-consistency loss. We input this image to the same encoder, which will produce a new code that we denote as (y 2, z 1) = E T (D T (y 2, z 1)). If the decoder correctly generates an image with attributes y 2, and the encoder is good at predicting the input image attributes, then y 2 should predict y 2. We use again the hinge loss to enforce this: DISPLAYFORM2 Here we could have used any random values instead of the sampled y 2. However, we found that sampling predictions from the data eases the task of the decoder, as it is given combinations of attributes that it has already seen. Despite this simplification, the decoder shows remarkable generalization to unseen values of the specified attributes y during evaluation. Finally, we add a cycle-consistency check on the unspecified part of the latent code, z 1 and z 1: DISPLAYFORM3 Encoder freezing. The training approach we just described presents a major pitfall. The reversed autoencoder could learn to replicate the input code (y 2, z 1) by encoding this information inside a latent image in whatever way it finds easier, that does not induce a natural attribute variation. To avoid this issue, a key ingredient of the procedure is to freeze the weights of the encoder when backpropagating L cyc1 and L cyc2. This forces the decoder to produce a naturally looking image so that the encoder correctly classifies its attributes. Global teacher loss. Overall, the global loss used to train the teacher is the sum of the five terms: DISPLAYFORM4 where λ i ∈ R, i = 1: 5 represent weights for each term in the sum. Details on how their values are found and how we optimize in practice are described in the next section. Ablation studies showing the contribution of each loss are shown in Section A.3 in the appendix. Student training. After the teacher is trained, we create a student autoencoder model with a larger dimension for the nuisance variables z and train it using only reconstruction and Jacobian supervision ( and ), as detailed in the next section. We implement both teacher and student autoencoders as Convolutional Neural Networks (CNN). Further architecture and implementation details are detailed in the Appendix. We train and evaluate our method on the standard CelebA dataset BID21, which contains 200,000 aligned faces of celebrities with 40 annotated attributes. The unspecified part of latent code (z) of the teacher autoencoder is implemented as a feature map of 512 channels of size 2×2. To encode the attributes part y, we concatenate an additional k = 40 channels. At the output of the encoder the values of these 40 channels are averaged, so the actual latent vector has k = 40 and d = 2048, dimensions for y and z respectively. The decoder uses a symmetrical architecture and, following BID19, the attribute prediction y is concatenated as constant channels to every feature map of the decoder. We perform grid search to find the values of the weights in FORMULA1 by training for 10 epochs and evaluating on a hold-out validation set. The values we used in the experiments in this paper are λ 1 = 10 2, λ 2 = 10 −1, λ 3 = 10 −1, λ 4 = 10 −4, λ 5 = 10 −5. At the beginning of the training of the teacher, the weights of the cycle-consistency losses λ 4 and λ 5 are set to 0, so the autoencoder is only trained for reconstruction (L rec), attribute prediction (L pred) and linear decorrelation (L cov). After 100 training epochs, we resume the training turning on L cyc1 and L cyc2 and training for another 100 epochs. At each iteration, we do the parameter updates in two separate steps. We first update for DISPLAYFORM0 Then, freezing the encoder, we do the update (only for the decoder), for DISPLAYFORM1 After the teacher autoencoder training is completed, we create the student model by appending new convolutional filters to the output of the encoder and the input of the decoder, so that the effective dimension of the latent code is increased. In this experiment, we first doubled the size of the latent code from d = 2048 to d = 4096 at the 200 th epoch and then from d = 4096 to d = 8192 at the 400 th epoch. Note that this is different to the experiment of Section 3, where we grew d by one unit at at time. We initialize the weights of the student with the weights of the teacher wherever possible. Then, we train the student using the reconstruction loss and the Jacobian loss as defined in Section 3, using λ y = 1, λ dif f = 50, and no prediction nor cycle-consistency loss (λ 2 = λ 4 = λ 5 = 0). The hyperparameters were found by quantitative and qualitative evaluation on a separate validation set. From CelebA, we use 162,770 images of size 256x256 for training and the rest for validation. All the figures in this paper show images from the validation set and were obtained using the same single model. For each model, we evaluated quantitatively how well the generated image is conditioned to the specified factors. To do this, for each image in the CelebA test set, we tried to flip each of the disentangled attributes, one at a time (e.g. eyeglasses/no eyeglasses). The flipping is done by setting the latent variable y i to −α · sign(y i), with α > 0 a multiplier to exaggerate the attribute, found in a separate validation set for each model (α = 40 for all models).To verify that the attribute was successfully flipped in the generated image, we used an external classifier trained to predict each of the attributes. We used the classifier provided by the authors of Fader Networks, which was trained directly on the same training split of the CelebA dataset. Table 2 and Figure 4 show the quantitative we obtained. Most notably, at approximately the same reconstruction performance, the student with Jacobian supervision is significantly better at flipping attributes than the student without it. With the Jacobian supervision, the student maintains almost the same disentangling and conditioning ability as the teacher. Note that these numbers could be higher if we carefully chose a different value of α for each attribute. To the best of our knowledge, Fader Networks BID19 constitutes the state-of-the-art in face image generation with continuous control of the facial attributes. For comparison, we trained Fader Networks models using the authors' implementation with d = 2048 and d = 8192 to disentangle the same number of attributes as our model (k = 40), but the training did not converge (using the same provided optimization hyperparameters). We conjecture that the adversarial discriminator acting on the latent code harms the reconstruction and makes the optimization unstable. Comparisons with these models are shown in Table 2 and in FIG4 in the appendix. We also show Table 2: Quantitative comparison of the disentanglement and reconstruction performance of the evaluated models in the facial attribute manipulation task. Disentanglement is measured as the ability to flip specified attributes by varying the corresponding latent unit. Figure 4: Disentanglement versus reconstruction trade-off for the facial attribute manipulation example (top-left is better). The disentangling score measures the ability to flip facial attributes by manipulating the corresponding latent variables.in FIG2 that our multiple-attribute model achieves similar performance to the single-attribute Fader Networks models provided by the authors. Finally, FIG1 shows the of manipulating 32 attributes for eight different subjects, using the student model with Jacobian supervision. Note that our model is designed to learn the 40 attributes, however in practice there are 8 of them which the model does not learn to manipulate, possibly because they are poorly represented in the dataset (e.g. sideburns, wearing necktie) or too difficult to generate (e.g. wearing hat, wearing earrings). A natural trade-off between disentanglement and reconstruction exists when learning image representations using autoencoder architectures. In this work, we showed that it is possible to overcome this trade-off by first learning a teacher model that is good at disentangling and then imposing the Jacobian of this model with respect to the disentangled variables to a student model that is good at reconstruction. The student model then becomes good at both disentangling and reconstruction. We showed two example applications of this idea. The first one was to progressively learn the principal factors of variation in a dataset, in an unsupervised manner. The second application is a generative model that is able to manipulate facial attributes in human faces. The ing model is able to manipulate one order of magnitude more facial attributes than state-of-the-art methods, while obtaining similar or superior visual , and requiring no adversarial training. For the autoencoder utilized for experiments in Section 3, we used the following architecture. For the encoder: DISPLAYFORM0 where F (I, O) indicates a fully connected layer with I inputs and O outputs. For the first teacher model (k = 2, d = 0), we also used BatchNorm after the encoder output. The decoder is the exact symmetric of the encoder, with a Tanh layer appended at the end. We used Adam with a learning rate of 3e −4, a batch size of 128 and weight decay coefficient 1e −6. Following BID19, we used convolutional blocks of Convolution-BatchNorm-ReLU layers and a geometric reduction in spatial resolution by using stride 2. The convolutional kernels are all of size 4×4 with padding of 1, and we use Leaky ReLU with slope 0.2. The input to the encoder is a 256×256 image. Denoting by k the number of attributes, the encoder architecture can be summarized as: DISPLAYFORM0 where C(f) indicates a convolutional block with f output channels. The decoder architecture can be summarized as: DISPLAYFORM1 where D(f) in this case indicates a deconvolutional block doing ×2 upsampling (using transposed convolutions, BatchNorm and ReLU) with f input channels. We trained all networks using Adam, with learning rate of 0.002, β 1 = 0.5 and β 2 = 0.999. We use a batch size of 128. TAB3 shows a comparison chart between the proposed and related methods. We applied the procedure described in Section 3 for progressive unsupervised learning of disentangled representations to the Street View House Numbers (SVHN) dataset BID23. The SVHN dataset contains 73,257 32×32 RGB images for training. For this experiment, the encoder architecture is: DISPLAYFORM0 Here, C(n) represents a convolutional block with n 3 × 3 filters and zero padding, ReLU activation function and average pooling. The decoder architecture is: DISPLAYFORM1 Here, D(n) represents a ×2 upconvolution block with n 4 × 4 filters and zero padding, ReLU activation function and average pooling. The latent code was started with k = 2 and d = 0 and progressively grown to k = 2, d = 16. Each stage was trained for 25 epochs. We used λ y = 0.025, λ dif f = 0.01. We used Adam with a learning rate of 3e − 4, a batch size of 128 and weight decay coefficient 1e − 6.The first teacher model (k = 2, d = 0) achieves a reconstruction MSE of 1.94e −2 and the final student model (k = 2, d = 16) a reconstruction MSE of 4.06e−3. FIG6 shows the two principal factors of variation learned by the first teacher model (corresponding to k = 2, d = 0). Contrary to the MNIST example of Section 3, here the two main factors of variation are not related to the digit class, but to the shading of the digit. The progressive growth of the latent code is carried on from d = 0 to d = 16. The following factors of variation are related to lighting, contrast and color (see FIG0). In this case, the unsupervised progressive method discovered factors that appear related to the digit class at the 9 th and 10 th steps of the progression. FIG0 shows how the digit class can be controlled by the student with d = 16 by varying these factors. Because of the Jacobian supervision, the student is able to control the digit class while maintaining the style of the digit. Finally, in FIG0 we show that the student also maintains control of the two main factors of variation discovered by the first teacher. Figure 10: Third, fourth and fifth factors of variation automatically discovered on SVHN. Each row corresponds to one factor and each column corresponds to one sample. Each factor is varied while maintaining the rest of the latent units fixed. FIG0: Factors of variation related to the center digit class appear to emerge on the 9th and 10th discovered factor during the unsupervised progressive procedure described in Section 3. Here we show how the student model with Jacobian supervision and d = 16 can be used to manipulate the digit class while approximately maintaining the style of the digit, by varying the latent units corresponding to those factors. The bottom row shows the original images (reconstructed by the autoencoder). All images are from the test set and were not seen during training. FIG0: Result of the student with Jacobian supervision (d = 16) when varying the two factors learned by the teacher FIG6, for four different images (whose reconstruction is shown on the bottom row). The conditioning related to shading is maintained. (Left to right: darker to lighter. Top to bottom: light color on the left to light color on the right.) All images are from the test set and were not seen during training.

Represent the short summary for finding the corresponding scientific article (long).

A method for learning image representations that are good for both disentangling factors of variation and obtaining faithful reconstructions.

represent the tldr for finding the corresponding scientific article

Example:
Provided: "We present a method to estimate collections of regression models in which each model is personalized to a single sample." Match: "When data arise from multiple latent subpopulations, machine learning frameworks typically estimate parameter values independently for each sub-population. In this paper, we propose to overcome these limits by considering samples as tasks in a multitask learning framework."

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: We propose a new notion of'non-linearity' of a network layer with respect to an input batch that is based on its proximity to a linear system, which is reflected in the non-negative rank of the activation matrix. We measure this non-linearity by applying non-negative factorization to the activation matrix. Considering batches of similar samples, we find that high non-linearity in deep layers is indicative of memorization. Furthermore, by applying our approach layer-by-layer, we find that the mechanism for memorization consists of distinct phases. We perform experiments on fully-connected and convolutional neural networks trained on several image and audio datasets. Our demonstrate that as an indicator for memorization, our technique can be used to perform early stopping. A fundamental challenge in machine learning is balancing the bias-variance tradeoff, where overly simple learning models underfit the data (suboptimal performance on the training data) and overly complex models are expected to overfit or memorize the data (perfect training set performance, but suboptimal test set performance). The latter direction of this tradeoff has come into question with the observation that deep neural networks do not memorize their training data despite having sufficient capacity to do so BID38, the explanation of which is a matter of much interest. Due to their convenient gradient properties and excellent performance in practice, rectified-linear units (ReLU) have been widely adopted and are now ubiquitous in the field of deep learning. In addition, the relative simplicity of this function (max(·, 0)) makes the analysis of ReLU networks more straight-forward than networks with other activation functions. We propose a new notion of'non-linearity' of a ReLU layer with respect to an input batch. We show that networks that generalize well have deep layers that are approximately linear with respect to batches of similar inputs. In contrast, networks that memorize their training data are highly nonlinear with respect to similar inputs, even in deep layers. Our method is based on the fact that the main source of non-linearity in ReLU networks is the threshold at zero. This thresholding determines the support of the ing activation matrix, which plays an important role in the analysis of non-negative matrices. As we discuss in Section 3, the non-negative rank of a matrix is constrained by the shape of the support, and is therefore indicative of the degree of non-linearity in a ReLU activation matrix with respect to the input. Although computing the non-negative rank is NP-hard , we can restrict it with approximate non-negative matrix factorization (NMF) BID20. Consequently, we propose to estimate the'non-linearity' of a ReLU layer with respect to an input batch by performing NMF on a grid over the approximation rank k, and measuring the impact on network performance. This procedure can be seen as measuring the robustness of a neural network to increasing compression of its activations. We therefore compare our NMF-based approach to two additional dimensionality reduction techniques, namely principal component analysis (PCA) and random ablations. We informally define memorization as the implicit learning of a rule that associates a specific sample (i.e., with index i) to a particular label (e.g., with index j). Such a rule does not benefit the network in terms of improving its performance on new data. We show that our NMF-based approach is extremely sensitive to memorization in neural networks. We report for a variety of neural network architectures trained on several image and audio datasets. We conduct a layer-by-layer analysis and our reveal interesting details on the internal mechanism of memorization in neural networks. Finally, as an indicator for memorization, we use our proposed measure to perform early stopping. The study of factors involved in the bias-variance tradeoff in learning models goes back several decades. Classical in statistical learning consider properties of learning models such as the VC-dimension BID33 and Rademacher complexity BID4. These properties give generalization bounds in terms of the capacity model to (over)fit data. When considering the vast capacity of deep neural networks, such bounds become irrelevant and fail to explain their ability to generalize well in practice BID38 BID5.More direct analyses have been done with respect to a specific setting of model parameters. For instance, BID3 showed that the number of weights in a network is less important compared to their scalar value (e.g. 2 -norm), and more recently BID5 presented a bound for deep neural networks based on the product of spectral norms of the network's weight matrices. BID0 showed that memorizing networks contain more information in their weights. Methods to explain generalization have been proposed that examine a network's robustness to perturbations BID13 BID6 BID14 BID26 BID22. These methods propose the notion of flatness of minima on the loss surface, assuming that perturbing the parameters without dramatically changing performance is an indicator of the generalization of a network. However, any reversible transformation, such as simple scaling, can arbitrarily manipulate the local flatness without affecting generalization BID8. The procedure we propose can be viewed as applying perturbations, albeit to activations and not parameters, and must address this concern. The perturbations we apply to activations account for magnitude, since they depend on a change of rank or non-negative rank of the activation matrix, a property which is robust to rescaling and similar reversible transformations. In contrast to the methods described thus far, which deal exclusively with the parameters of the model, methods have been developed that account for the role of the data distribution. BID23 proposed to use the Fisher-Rao norm, which uses the geometry of the data distribution to weigh the contribution of different model parameters. The empirical studies of BID24 and BID27 explore robustness to specific types of noise. The former uses Gaussian noise and masking noise injected into hidden activations, while the latter interpolates between input samples to study network behavior on and off the data manifold. In both cases, robustness to noise proved a reliable indicator for good generalization. Additionally, BID1 derive generalization bounds in terms of robustness to noise. Our experimental setup is reminiscent of BID24 in that both methods apply a form of compression to hidden activations and test for robustness to this type of noise. Specifically, they set random axis-aligned directions in feature space to zero which can be viewed as a crude form of dimensionality reduction, i.e., by simply removing canonical dimensions. In our experiments we refer to this method as random ablations. Our show that robustness to NMF compression is much more correlated with low memorization/high generalization than robustness to random ablations. BID2 have also studied various empirical aspect of memorization. As a dimensionality reduction technique, NMF has gained popularity due to its producing meaningful factorizations that lend themselves to qualitative interpretation across various domains such as document clustering BID37, audio source separation BID11, and face recognition BID12. In the context of deep convolutional neural networks, BID7 applied NMF to the activations of an image classifier and showed that the gives a decomposition into semantic parts, which benefits from the transformation invariance learned by the neural network. Consider a ReLU layer of a neural network, parameterized by a weight matrix W ∈ R m×q. For a batch of n inputs X ∈ R n×m, we compute the layer activation matrix A as follows: DISPLAYFORM0 where R + are the non-negative reals. We omit the bias term for notational convenience. The processing of a single input x by a ReLU network is equivalent to sampling a sub-network that is linear with respect to the sample BID35. This could be accomplished by simply setting to zero the columns of each W whose dot product with the input is negative (and would thus be set to zero by ReLU), and then removing the thresholding. Extending this notion to a batch of several input samples to a ReLU layer, suppose the samples are sufficiently close to each other such that they all share the same ReLU mask m ∈ {0, 1} q. In this case, we may say that the layer is linear with respect to its input batch. This is because, for the entire batch, instead of using ReLU, we could zero out a subset of columns and obtain a linear system, i.e., A = XW diag(m).For an activation matrix A (Equation 1), we consider the support M = supp(A), which we describe as a binary 0/1 matrix where M i,j = 1 where A i,j > 0. Because A is a ReLU activation matrix, the structure of M is mainly determined by the thresholding at zero.2 Because thresholding is the main source of non-linearity in a ReLU network, the support takes on a special meaning in this case. We want to characterize how close to being linear a layer is with respect to its input X by examining the support of the ing activations M. If all the rows of M are identical to a unique vector m, we can say the layer is completely linear with respect to X. In general, the'simpler' the support M, the closer to linearity the layer. One measure that captures this idea is the rectangle cover number of a matrix, rc(M), an important quantity in the study of communication complexity BID17. Also known as the Boolean rank, rc(M) is the smallest number r for which there exist binary matrices U B ∈ {0, 1} n×r, V B ∈ {0, 1} r×q such that their Boolean matrix multiplication satisfies M = U B V B. As a complexity measure for ReLU activations, rc(M) = 1 means the layer is linear with respect to its input, and higher values rc(M) imply increasing non-linearity. This is visualized in FIG0.Intuitively, imagine having to fit a layer with'ReLU switches', each of which controls a subset of weight matrix columns. In the linear case, one switch would suffice to describe the data. In the most non-linear case, we would require a switch for every column, which is also the maximal value of rc(M).Because computing the rectangle cover number rc(M) is complex, several approximations and bounds to it have been studied BID9. For the support of a non-negative matrix, a well-known upper-bound is: DISPLAYFORM0 where rank + (A) is the non-negative rank of A BID10 that is defined as the smallest number k such that there exist non-negative matrices DISPLAYFORM1 Similar to the rectangle cover number, the non-negative rank is hard-constrained by the combinatorial arrangement of supp(A), but additionally accounts for the actual value in the non-zero entries of A.While computing rank + (A) is not easier than computing rc(supp(A)), we can restrict it by performing approximate non-negative matrix factorization (NMF). For a given non-negative rank constraint k, NMF solves for: DISPLAYFORM0 with U +, V + as defined above. The U + V + =Ã k ≈ A is the closest matrix to A under the Frobenius norm that has rank + at most k. Consequently, we propose to estimate the'linearity' of a ReLU layer with respect to a batch of similar inputs by performing NMF on a grid over the non-negative rank k, and measuring the impact on network performance by observing the change in the prediction (output layer) as we change k. This procedure also addresses the fact that in practice network activations tend to be noisy, whereas supp(A) is not robust to noise, i.e., A i,j = > 0 → M i,j = 1 even for very small.Concretely, if we let A i be the activation matrix at layer i, during the forward pass, we replace the feature activations of one or several layers with their rank k NMF approximations: DISPLAYFORM1 For convolutional networks, we first reshape the tensor of feature maps from n × q × h × w to (n · h · w) × q, i.e., we flatten the batch (n) and spatial dimensions (h, w) to form a matrix with q columns, where q is the number of channels in that layer. We then inversely reshape the approximated features to continue forward propagation through the network. We now characterize the input batch, with respect to which we would like to measure layer linearity. Informally, the goal of training is to cluster together input samples that have similar (or identical) output labels,while separating them from samples of other labels. In the context of classification then, we expect therefore that from a certain depth and onward, samples of the same class will have similar activations, and thus a simpler support. In other words, while a network may exploit flexible non-linear structure to separate different classes, we expect that with respect to a single class, deep layers are approximately linear.. We analyze each layer by applying NMF compression to its activation matrix with increasing rank k, while observing the impact on classification performance. In (a) and (b) we show the k vs. accuracy curves at an layers of different depth. We can immediately see that in deep layers, networks with high memorization are significantly less robust to NMF compression, indicating higher degrees of non-linearity. Furthermore, networks trained on fully randomized labels (p = 1) behave differently than networks with partial or no randomization. By summarizing each curve in (a) and (b) by its area under the curve (AuC), we show in (c) a birds-eye view over all layers. All networks with p < 1 pass through distinct phases consisting of a feature extraction phase until conv3_1, followed by a memorization phase until conv4_2, followed by a final clustering phase. Interestingly, the case p = 1 shifts the process into earlier layers, explaining why layer-by-layer it appears as an outlier. When single-class batches are not approximately linear even in deep layers, i.e., activations are not clustered within a few linear regions, we empirically show in the next section that this behavior is indicative of memorization. We start by studying networks that have been forced into different levels of memorization due to label randomization applied to their training set BID38. The level of induced memorization is controlled by setting a probability p for a training label to be randomized, i.e., p = 0 is the unmodified dataset and p = 1 gives fully random labels. Note that the capacity of these networks is sufficiently large such that the training accuracy is 1 in all cases, regardless of the value of p. As such, we use batches of training data and observe how accuracy drop from 1 to constant prediction as we increase the level of compression. In all experiments, sampling single-class batches is done with respect to the label used for training (i.e., the random label if p > 0). We sample batches stochastically (up to the label), have found all methods discussed below to be robust to the batch size (e.g., 20-100). In all our experiments we set the batch size to 50.We perform experimental evaluations on several image datasets, namely CIFAR-10 (BID18), Fashion-MNIST BID36, SVHN BID25 ImageNet , as well as on the Urban Sounds audio classification dataset BID30. We use a fully-connected network for Fashion-MNIST and various CNN architectures for the others, which we describe in more detail the appendix. We start by analyzing the layers of an 11-layer CNN trained on CIFAR-10. We sampled 10 batches (one batch per class) of 50 images, and compressed the activation matrix at each layer individually down to various values of the non-negative rank. We then measured classification accuracy of the prediction. In this analysis we report average for 60 neural networks, ten networks (with different random initializations) trained per randomization level p. In FIG1 (a) and (b) we show k vs. accuracy curves of networks trained with increasing levels of label randomization, at an early layer (conv2_1) and a deep layer (conv4_1) respectively. We can immediately see that networks trained on fully randomized labels (p = 1) behave differently than networks with partial or no randomization. Furthermore, note that in deep layers, memorizing networks are significantly less robust to NMF compression, i.e., their activations posses a high nonnegative rank, which indicates high non-linearity with respect to the input, as discussed in Section 3.2. We can characterize each curve in (a) and (b) with a single number, its area under the curve (AuC). This allows us in FIG1 (c) to generate a single figure for all layers. Networks with p < 1 display a similar feed-forward trend up until layer conv3_1. Since these networks differ from each other in no way other than the level of label randomization on the training data, we hypothesize this to be a generic feature extraction phase common to all of them. In the next phase, until conv4_2, we see a big difference between networks, such that more memorization (higher p) is correlated with lower AuC, i.e., higher non-negative rank and hence non-linearity of those layers with respect to single-class batches. We therefore localize memorization to these layers. Lastly, the phase only of conv4_3 is where samples of the same class are clustered together, right before the final 10-dimensional classification layer (which is not shown). This final phase is in accordance with the premise that regardless of randomization level p, all of these networks achieve perfect training set accuracy. Interestingly, setting p = 1 shifts the process to earlier layers, explaining why layer-bylayer this case appears as an outlier. We compare different compression techniques with regards to detecting memorization. Given our choice of solving NMF under the Frobenius norm (Equation 3), a natural method to compare against is principal component analysis (PCA), which optimizes under the same norm but without the nonnegativity constraint on its factors. We also consider random ablations, i.e., setting a random subset of columns in the activation matrix to zero, since this technique has been used previously to detect memorization BID24.Rather than choosing a single layer, we sequentially apply compression to several layers. We target the final convolutional blocks of our CNNs, all of which contain three layers, each of which consists of 512 channels. In fully-connected networks, we applied compression to all layers. In FIG2 we give for the CIFAR-10 dataset, which confirm that NMF compression is indeed more sensitive to memorization, due to the properties of the non-negative rank discussed in Section 3.2. PCA, which is less constrained, is more'efficient' at compressing the activations, but is in turn less discriminative with respect to the level of memorization. Finally, we confirm that robustness to random ablations correlates with less memorization, however less so than NMF. It should be noted that NMF does show more variance than the other two methods, and incurs a higher computational cost, as discussed in section 6.6 in the appendix. In FIG3 we show additional for single-class NMF on three additional datasets and network architectures (described in the appendix), including a fully-connected network for Fashion-MNIST. The in (d) and (e), of applying PCA and NMF to multi-class batches, show that such batches produce activations with higher rank or non-negative rank compared to single-class batches. This is a of the network trying to separate samples of different labels. We have shown for networks forced into memorization due to label randomization. In this section we show our technique is useful for predicting good generalization in a more realistic setting, without artificial noise. In addition to the experiments below, we refer the reader to Section 6.3 in the appendix, where predict per-class generalization of a pre-trained VGG-19 network on a ImageNet classes. We trained 96 CNN classifiers on CIFAR-10, over a grid of hyper-parameter values for the batch size, weight decay and optimization algorithm, SGD vs. ADAM BID16. Following the same procedure as above, for each of the three methods, NMF, PCA, or random ablations, we computed the k vs. accuracy curves for each network, targeting its final convolutional block. In Figure 5 we compare the area under the curve (AuC) of each curve with the average generalization error on the test set. While all three methods show correlation with generalization error, NMF is most correlated with a Pearson correlation of -0.82, followed by PCA with -.064 and random ablation with -0.61. We test whether our method can detect memorization during training. Doing so would allow us to perform early stopping, i.e., stop training as memorization begins to decrease generalization. We trained CNNs on CIFAR-10 with the original labels. Each network was trained for 10K batches with a batch size of 100. We recorded the test set error every 250 batches, and tested the non-linearity of the deepest three convolutional layers using our NMF-based approach with a coarse grid on k. As before, we compute the area under each k vs. accuracy curve as in FIG2. Finally, we also computed the area under the curve produced by random ablations. Results of two instances are shown in Figure 6 (a) and (b). In these figures we compare the test loss against our single-class NMF method and random ablations. We smooth the plots using a radius of two epochs to reduce the noise. The matching-color dashed lines mark the local minima of the test loss in as well as the location of the first local maxima of the NMF and random ablation AuC curves after smoothing has been applied. We notice that the test loss minima align almost precisely with the maximum NMF AuC. We further confirm this behavior in Figure 6 (c), where we compare the stopping times of NMF and the random ablations method against the best test loss over 10 different runs. We have introduced a notion of a ReLU layer's non-linearity with respect to an input batch, which is based on its proximity to a linear system. We measure this property indirectly via NMF applied to deep activations of single-class batches. While more analysis is required before definite guarantees could be given, we find that our approach is successful in detecting memorization and generalization across a variety of neural network architectures and datasets. The exact architectures we used for each dataset are given in Table 1. We denote a linear or convolutional layer followed by a ReLU as Linear + and Conv +, respectively. It is interesting to study the impact of ablating the activation in the directions found by NMF and PCA by forward propagating the residual, i.e., DISPLAYFORM0 This is interesting because in the case of PCA, for instance, the top k directions are those that capture most of the variance in the activation matrix, and presumably the k directions found by NMF are of similar importance. This is not true for the random ablations, where the ablated directions are of no special importance. In Figure 7 we see that networks with no induced memorization that are most vulnerable to ablation of NMF and PCA direction. In other words, while non-memorizing networks are more robust to random ablations, they are not robust to ablations of specific important directions. This is in contrast to the interpretation of Morcos et al. FORMULA0 The VGG-19 model BID31, trained on ImageNet , is known for its good generalization ability, as evident by its widespread use as a general feature extractor. We use a pre-trained model here as an example of a well-generalizing network and analyze it with our method. We apply NMF compression to the three deepest convolutional layer, on activations of both singleclass batches and multi-class batches. We select 50 random classes from ImageNet and gather batches of 50 training samples from each class. In Figure 8, shown in blue, NMF applied to single-class batches, has a denoising effect and improves over the baseline accuracy of the batch, shown as a dashed line. As the constraint on k is relaxed, that accuracy drops back to its baseline level. We contrast this behavior with the one shown in green when using multi-class batches. Here, only when k is large do we regain baseline accuracy, and sensitivity to ablation is similarly diminished. This is due to the critical role non-linearity plays in separating the different classes. Ablating the NMF directions dramatically reduces classification accuracy. Finally, in Figure 8 (c) we show there is a significant per-class correlation (Pearson r = 0.78) between NMF AuC and accuracy on test accuracy on batches from the ImageNet test set. In this experiment we use NMF-based early stopping to improve neural network accuracy in the context of few-shot learning, i.e., learning with very few samples per output label. We choose this setting since it is representative of data scarcity, where one would like to use all available data for training, rather than holding some out for validation. We demonstrate this on the case of MNIST BID19 digits with only 2 samples per class, which in a training set of 20 samples. For early stopping with NMF, we set a very simple grid over k, a single point at k = 1. Thau NMf AuC thus simply becomes the training accuracy when compressed with NMF k = 1.In FIG6 it is evident that training set accuracy with NMF k = 1 shows similar gradients as the test set accuracy. Based on this observation, as before, we extract the first peak of the smoothed NMF curve, and stop there. Results are shown in Table 2 for 10 runs with randomly sampled training sets. We compare accuracy at our early stopping point to the best test set accuracy detected throughout the run (Best case), the average test set accuracy where the training set accuracy is 1 (Average case), and similarly the lowest test set accuracy where the set accuracy is 1 (Worst case).: NMF early stopping for few-shot learning of MNIST digits with only 20 samples. By observing the training set accuracy under NMF k = 1 compression, we are able to correctly guess the gradient of the test set accuracy. We use this to perform early stopping with the simple heuristic of stopping at the first peak, which leads to improved accuracy as shown in Tabel 2. Table 2: NMF early stopping for few-shot learning of MNIST digits with only 20 samples. By observing the training set accuracy under NMF k = 1 compression, we are able to correctly guess the gradient of the test set accuracy. Early stopping at the first peak consistently improves accuracy. In the first row of Table 2 we see that on average our method significantly improves over not using early stopping, and is on par with a recently proposed method specifically deigned for few-short learning. Furthermore, in the last two rows we show that per sampled training set, early stopping consistently improves accuracy. In FIG1 we show for every layer the area under the curve (AuC) of its k vs. classification accuracy curve. However in addition to the accuracy, the NMF reconstruction itself is also a quantity of interest. There main difficulty involved with interpreting the NMF error is scale. The error depends on the magnitude of the activations, which varies across networks, layers and even channels. In FIG0 Layer-by-layer view of (a) raw and (b) normalized NMF reconstruction errors, which NMF is trying to minimize. As we again notice the outlier behavior of networks trained with label randomization p = 1, in (c) we localize the transition between the two regimes to around p = 0.9. approximate with depth, with an interesting interaction between the memorization level and depth. The error in absolute terms echo the accuracy curve, with p = 1 again presenting outlier behavior. Returning to accuracy measurement as in 2 (c), sampling p more densely reveals in FIG0 (c) that a phase shift occurs around p = 0.9, where networks "shift" their memorization to earlier layers. Applying NMF compression to large matrices naturally incurs certain overhead. We find, however, that our implementation of the multiplicative update algorithm BID21 runs in reasonable time thanks to GPU acceleration. We report timing for typical batch used for VGG-19, i.e., 100 samples of 224×224 color images. At layer conv5_4 activations form a tensor of size 100 × 14 × 14, which we flatten to a matrix of size 19600 × 512. In FIG0 we show the timing curve for this batch as we increase k, using an NVIDIA Titan X card. As can be seen, at k = 500 processing of the batch to convergence requires 197 milliseconds on average. The final runtime depends on the number of classes sampled, and the granularity of the grid over k. We found our measurements to be robust to heavy subsampling of both.

represent the short summary for finding the corresponding scientific article (long)

We use the non-negative rank of ReLU activation matrices as a complexity measure and show it (negatively) correlates with good generalization.

Represent the TLDR for finding the corresponding scientific article.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.