Question: Applying reinforcement learning (RL) to real-world problems will require reasoning about action-reward correlation over long time horizons. Hierarchical reinforcement learning (HRL) methods handle this by dividing the task into hierarchies, often with hand-tuned network structure or pre-defined subgoals. We propose a novel HRL framework TAIC, which learns the temporal abstraction from past experience or expert demonstrations without task-specific knowledge. We formulate the temporal abstraction problem as learning latent representations of action sequences and present a novel approach of regularizing the latent space by adding information-theoretic constraints. Specifically, we maximize the mutual information between the latent variables and the state changes. A visualization of the latent space demonstrates that our algorithm learns an effective abstraction of the long action sequences. The learned abstraction allows us to learn new tasks on higher level more efficiently. We convey a significant speedup in convergence over benchmark learning problems. These demonstrate that learning temporal abstractions is an effective technique in increasing the convergence rate and sample efficiency of RL algorithms. Reinforcement learning (RL) has been successfully applied to many different tasks . However, applying it to real-world tasks remains a challenging problem, mainly due to the large search space and sparse reward signals. In order to solve this, many research efforts have been focused on the hierarchical reinforcement learning (HRL), which decomposes an RL problem into sub-goals. By solving the sub-goals, low-level actions are composed into high-level temporal abstractions. In this way, the size of the searching space is decreased exponentially. However, the HRL often requires explicitly specifying task structures or sub-goals . How to learn those task structures or temporal abstractions automatically is still an active studying area. Many different strategies are proposed for automatically discovering the task hierarchy or learning the temporal abstraction. Some early studies try to find sub-goals or critical states based on statistic methods (; ;). More recent work seeks to learn the temporal abstraction with deep learning (; ; a). However, many of these methods still require a predefined hierarchical policy structure (e.g. the number of sub-policies), or need some degree of task-specific knowledge (e.g. hand-crafted reward function). We present a general HRL framework TAIC (Temporal Abstraction with Information-theoretic Constraints), which allows an agent to learn the temporal abstraction from past experiences or expert demonstrations without task-specific knowledge. Built upon the ideas of options framework and motor skills , we formulate the temporal abstraction problem as learning a latent representation of action sequences. In order to obtain good latent representations, we propose a novel approach to regularize the latent space by using information-theoretic constraints. The learned abstract representations of action sequences (we called options) allow us to do RL at a higher level, and easily transfer the knowledge between different tasks. Our contributions are: 1) We formulate the temporal abstraction problem as learning a latent representation of action sequences. Motivated by works using Recurrent Variational AutoEncoders (RVAE) to model sequential data in neural language processing (NLP) and other areas , we employ RVAE to perform temporal abstraction in RL. 2) We propose a regularization approach on the option space. It constrains the option to encode more information about its consequence (how the option changes the states). We present both theoretical derivations and practical solutions. 3) We show in the experiments that our learned temporal abstraction conveys meaningful information and benefit the RL training. In addition, the proposed framework provides an efficient tool for transferring knowledge between tasks. HRL is a long-standing study topic. proposed HRL with temporal abstraction in early 1990s. proposed a system in which recurrent NNs generate sequences of sub-goals, a evaluator NN predicts the rewards of going from start to goal and a RL machine tries to use such sub-goal sequences to achieve final goals . The options framework is a popular formulation for considering the problem with a two-level hierarchy . A sequence of primitive actions are converted into options. The traditional Markov Decision Process (MDP) problem is extended into semi-MDP with the use of options. Parr developed an approach called Hierarchies of Abstract Machines to calculate hierarchically structured MDP policies . Dietterich proposed MAXQ method, which performs value function decomposition over a given task structure . These early work assume the task hierarchy is predefined by human experts. For the automatic task decomposition problem, many methods try to find sub-goals or critical states based on statistic methods (; ;). These methods cannot handle continuous control problem. More recent work seeks to learn the temporal abstraction with deep learning (; ; ; a;). developed a two-layer hierarchy of policies, including one meta-policy and several primitive policies. defined the policy sketches, which annotate tasks with sequences of sub-tasks, and learn the sub-tasks and upper-level tasks jointly. presented hierarchical-DQN, which integrates hierarchical action-value functions with intrinsic reward based on predefined sub-goals. combined the options framework with policy gradient and proposed an option-critic architecture. These methods either require a predefined hierarchical policy structure (e.g. the number of sub-policies) or need to specify sub-goals. We propose a framework that allows learning temporal abstraction without predefined sub-goals. The idea of learning latent representation for HRL has been proposed before ), but they did not learn from action sequences. The SeCTAR algorithm learns a latent representation from from state sequences in trajectories using RVAE . Macro-action methods share the similar idea of combining sequences of actions into options . However, existing literature only combines primitive actions, while our work learns an abstraction, which we will show later is more beneficial for HRL training. Our work is related to Fabius & van;; , since they also utilize RVAE to encode sequential data. In this paper, we apply RVAE to model the abstraction of action sequences. We consider a MDP problem M = (S, A, P, R, γ), where S is the set of states, A is the action set, P: (S, A) → S is the transition model, R: (S, A) → R is the reward function, γ is the discount factor. presents the option framework, where an option is formulated as a tuple (I, π, β) in which I ∈ S is the initiation set, π is the sub-policy and β: S → is the termination condition. The option framework and its followers either manually designed or learned a finite set of options. The number of option becomes a hyperparameter, and each option in the set is usually independent with one another. We propose the TAIC framework, in which we model the option o as a continuous random variable, which denotes the latent representation of a sequence of actions. Furthermore, following the option framework we define (I(o), π(o), β(s, o)) as the function of the random variable o. I(o) is the initiation condition, which is assumed to be the entire state space S for simplification. We define the function π(o) as the sub-policy that maps the latent variable o to a sequence of actions. The termination condition β(s, o) controls the temporal length of the option, and will be detailed later. This modification of the option definition brings important difference. In the original option framework, each option represents one sub-policy. Above those sub-policies, there is a meta-policy π Ω that acts as a classifier, choosing one of the sub-policies at a longer time horizon. In contrast, the TAIC framework specifies the option o as a continuous random variable, which could have an infinite number of values. Sub-policy is defined as a function over the random variable. Our meta-policy π Ω outputs directly the random variable o. This changes the framework from discrete options to continuous options, and allows us to put constraints on the option space. So that the options with similar consequences become closer in the option space. Given a set of past experiences Λ = {τ 0, τ 1, ..., τ m}, where τ i ∈ Λ is one trajectory {s 0, a 0, r 0, s 1, a 1, r 1, ..., s k, a k, r k}, which comes from the agent's past experience or expert demonstration, our problem is to learn a temporal abstraction o, and the corresponding, so that the o should be able to apply to the RL task, and improve the training efficiency. We consider the problem of learning latent representations o from action sequences {a 0...k0, a 0...k1, ...}. To model the sequential data with variable length, we use the recurrent autoencoder (AE), which has been used intensively in natural language processing (NLP) and other sequential modeling tasks . Specifically, we deploy recurrent variational auto-encoders (RVAE) (; Fabius & van), because it is empirically shown to be better at finding hidden features of the inputs . In general, we would like to calculate the posterior p(o|a 0...k), where the option o captures the intrinsic features of the action sequences. The RVAE solves this by approximating the true posterior by q(o|a 0...k) and then optimizing a lower bound on the log-likelihood (Fabius & van). The log-likelihood of an action sequence a 0...k can be written as: where KL(p|q) is the Kullback-Leibler divergence, and the L(o) is the evidence lower bound: In order to find a good approximation of p(o|a 0...k) with q(o|a 0...k), namely minimize KL(q(o|a 0...k)||p(o|a 0...k)), we need to maximize the evidence lower bound L(o). The conditional distributions q(o|a 0...k) and p(a 0...k |o) are approximated by two networks, a recurrent encoder Following the common VAE formulation, we model q(o) with the Gaussian distribution, and the training objective is to minimize a reconstruction loss and a KL loss : Specifically, the encoder E takes in an action sequence a 0...k, and outputs the option represented by two vectors o µ and o σ, namely the mean and standard deviation in the Gaussian distribution. The L KL is the KL divergence between this output and the prior Gaussian distribution N. The decoder D takes in the sample of option o, and outputsâ 0...k. The L Recons is defined as the mean-square-error between a 0...k andâ 0...k. The reconstruction loss in the RVAE setting suggests it encodes the action sequences with respect to the L2 distance in action space. Figure 1 shows a simple example in 2D navigation task. Two ) 0...3 have big L2 distance in action space, but they have the same consequence (both move from state s 0 to s 1). In favor of upper-level task solver, they should be encoded closely together in the option space. Imaging the navigation task in real life, in order to reach the door you could follow different paths (avoiding some dynamic obstacles). Those different sequences all have the same coding in your brain "go reach the door". In contrast to precisely reconstructing the action sequences, our goal is to extract the latent variable capturing the information which could benefit RL training. Intuitively speaking, the option should encode the consequence of action sequence, namely how the sequence changes the state. Formally, we maximize the mutual information between the option and state changes. We decompose it into two terms. The first term is to maximize the mutual information between the option and two states (states before and after a sequence of actions): On the other hand, the option should encode the state change, and be decoupled from particular start and end states. For example, navigating to the door from this room should be represented similarly as navigating to the door at another room. This is formulated as minimizing the mutual information between the option and the individual start and end state: Following the definition of the mutual information and the chain rule of the conditional entropy, Equation 4 is transformed into minimizing the summation of two conditional entropy: Similarly, Equation 5 is equivalent to maximizing the conditional entropy of s/s given o: Notice Equation 6 and Equation 7 conflicts with each other, which is also very intuitive. Forcing the option to be irrelevant to the start and end state would make it harder to encode the information of state changes. This is a trade-off depending on how much you want the code to be state-independent. Combining Equation 6 and Equation 7, we obtain our final constraints: Figure 2: System architecture. E, D, F, P and RL are implemented with neural networks. The gradients of three losses L rvae, L adv and L pred will be back-propagated through the encoder E, and regularize the option. Note the gradient of L adv will change sign before back-propagating to E. Those two constraints are also approximated with neural networks. For Equation 8, we utilize a predictor network P: (s, o) → s, which predicts the end state s given the start state s and the option o. To optimize the constraint, we minimize the predictive loss L pred, and backpropagate the gradients into the encoder E. In order to minimize the log-likelihood in constraint Equation 9, we borrow the technology from the adversarial training . We utilize another network F: o → (s, s), which tries to recover the start state and the end state given the option code. The encoder E and the state estimator F are trained competitively with each other. F acts like a discriminator, trained in minimization of the loss of estimating the states L adv, while E is trained with the opposite gradients that try to maximize the loss L adv. Thus, E will be pushed to encode option in a way that F is not able to recover the start and end states. In this way, the encode E is regularized by these extra gradients. In practise, all of these networks are trained jointly on the experience set Λ = {τ 0, τ 1, ..., τ m}. As Figure 2 shows, we have 4 networks (E, D, F and P) collaborating and competing (note RL is not updated in the option learning process). The learned latent representation o is a temporal abstraction that captures the execution consequence of the action sequences. After learning the option, we can train the agent at a higher level. The HRL policy outputs an option, which is then decoded to a sequence of actions by the decoder D. In order to apply the learned option to HRL training, we need to consider the termination condition β(s, o). We implement and compare three different methods: Fix-length, Term-output and Term-predict. In the simplest setting, we learn the option from the action sequence with fixed length N. The decoder D terminates the output after N actions, and then the HRL policy will output another option. This termination condition (referred as Fix-len later) does not depend on the option and state. This straightforward method is easy to implement. We will show in the experiment section that for most tasks, even this naive implementation would bring benefit to the HRL training. In order to learn the option code of arbitrary sequence length, we add another output to the decoder D. At each time step t, D outputs an action a t and a termination signal c t, which determines whether this is the last action in the sequence (Figure 3 (a) ). The termination output is a 2-class classifier, trained with supervised cross-entropy loss. Note that this termination condition (referred to as Term-output) is only dependent on the option. Allowing the RVAE to encode various lengths of sequence provides the HRL with more choices. For example, in states that are stable and safe, the HRL can choose longer sequences; while in unstable backpropagate L pred i:= i + 1 8: end while 9:â 0.. states, it can operate cautiously with shorter sequences. However, this termination condition is not dependent on the state, so it cannot respond to sudden changes in the state. In this setting, we utilize the predictor network P to decide the length of action sequences (Figure 3 (b) ). The intuition is that the predictor network P also acts as a world model , which models the environment dynamics. When things are within expectation, the decoder D can go ahead and output longer sequence. When the agent encounters unfamiliar states, D should become more cautious and output shorter sequence. This termination condition (referred to as Termpredict) depends on both the option and the state. During training, we set a threshold δ on the prediction loss L pred. When L pred is bigger than δ, the encoder D terminates the output, then the HRL policy will output a new option for subsequent actions. The training algorithm is detailed in Algorithm 1. The option learned in this way is supposed to be more robust to state changes. However, this method relies on a good predictor. The length of the sequence is dependent on the hyper parameter δ. Given the option, we employ Semi-MDP framework to train an HRL agent at a higher level. Because the option is in continuous space, we use policy gradient algorithm and derive the algorithm over options. Note that although our option is continuous, our temporal abstraction could be applied to discrete problems, as long as the RVAE outputs discrete actions.;. The gradient over the parameters has the form: where the π(o|s; θ) is the high-level policy planning in state-option space. The advantage function A(s t, o t) is designed as the accumulative reward subtracting a baseline function: where R(s t, o t) is the cumulative discounted reward from time t to the future, and the baseline function V (s) is updated with the TD algorithm (assuming the upcoming options are of length t 1, t 2, ...): where r t→t+t1 is the discounted cumulative reward during executing o t between t and t + t 1: In the experiments, we employ PPO algorithm. In general, the policy gradient over option and the policy gradient over action are similar. Most of the existing RL algorithms such as TRPO, TNPG and SAC (b) are applicable. We first consider a 2D navigation task for proof of the concept. Then we apply our temporal abstraction framework to robotic control tasks in MuJoCo . At last we will move to more challenging tasks, which are also used by Haarnoja et al. (2018a) and , and are hard to solve by non-hierarchical methods. The experiments in this section are generally performed in three steps: 1) collecting the experiences using a flat PPO agent; 2) learning the option based on the TAIC algorithm; 3) training HRL based on the option. We implement a one-layer LSTM with 64 hidden units for both the E and D, and three-layer MLPs for the RL, F and P. We use learning rate 0.01 for the E, D and P, learning rate 0.0003 for the RL, 0.001 for F. We also balance multiple losses with different weights: λ kl is used for the L kl in RVAE; λ adv is used for L adv for F. We first test our framework using a 2D navigation task, which is a toy example that allows us easily visualize the option we learned from the experience. As Figure 4 (a) shows, our environment is a 10m by 10m room with obstacles. Both the state and the action are continuous, represented by (x, y) and (v x, v y) respectively. The goal is to navigate from the start location (orange circle) to the goal location (green circle). The reward for reaching the goal is 100, and -1 otherwise. We collect the experience using a flat PPO agent, running on multiple tasks with random start and goal location. Then we train an RVAE network using the collected action sequences. Figure 5 (a) shows the interpolation between two options. We first randomly sample two action sequences from the testing set (shown in red dash lines), and encode them into two options using the network E. We do a linear interpolation between these two options and then decode those options into a set of action sequences using network D. The first thing we notice is that the RVAE nicely capturing the direction and curvature of the sequences and usually outputs a smoother version (solid lines) of the original sequence (red dash lines). Further, the interpolations between two sequences smoothly transfer from one to the other, denoting that we have a smooth option space. In Figure 5 (b), we visualize how each dimension in the option space encodes different information. The red dash line shows an action sequence sampled from the testing set. This action sequence is encoded into an option which is a vector with 6 real values. We modify one dimension and decode the options into action sequences shown with the solid lines. We notice the first two dimensions control the ending point of the action sequence, and dimension 3 and 4 control the curvature, dimension 5 and 6 control more subtle properties. Note that not all the trials of training get the same . Sometimes the properties are mixed, but Figure 5 illustrates one of the most common . Next, we evaluate the ability to recover from a sudden change in the environment. and also presented a similar experiment. Initially the environment is set up as Figure 4 (a) left. After 1000 episodes, an obstacle blocks the shortest path, and creates a trap (Figure 4 (a) right). We reset the standard deviation output of the PPO algorithm to encourage exploration. The flat RL is hard to recover from this situation since the exploration at each time step still gives a similar trajectory, which leads to the trap. While the exploration in the option space in more diverse trajectories. We average 10 repeated experiments for both settings. As can be seen in Figure 5 (c), the TAIC recovers from the environment changes for all 10 trials. While the flat RL agent find the goal in 20% trials. In the robotic control domain, we utilize the MuJoCo tasks in the OpenAI Gym environment . The goal is to learn a control policy π: S → A, so that the robot runs forward as fast as possible (Figure 4 (b) ). Both the state and the action are vectors of continuous values. We present two ways of evaluating the TAIC framework. Firstly, we qualitatively evaluate the learned option by visualize the option space. Secondly, we apply the learned option to the HRL training and compare the performance. Evaluating the high dimensional option space is difficult. We observe that the losses in the option learning process (L rvae, L pred and L adv) could not reveal the underlying structure of the option space. We propose a visualization method that provides a qualitative evaluation of the option space. We now consider how to visualize the correlation between the option and state change. We randomly sample the options ). The options and the end states are first converted to 2D vectors using t-SNE , respectively. The 2D vector of the end states are associated to the location of the points in the figure, while the color of the points represents the distribution of the options. This is to say, if the option distribution is more coupled with the state change, the points will be arranged better with respect to the color. As we can see from Figure 6, with information-theoretic constraints the options and state changes become more correlated. We evaluate the option on four benchmark tasks in simulation. Figure 7 compares the HRL performance with different termination conditions. We observe the HRL using TAIC outperform the flat PPO in 3 out of 4 tasks. In the HarfCheetah and the Ant task, the curve of TAIC starts higher (because the random policy over the option space already gets positive rewards) and converges faster. One thing should be notice is that the HRL policy is updated in a much lower frequency than the flat RL, because the options are executed in multiple time steps (a typical average length is 5 time steps). In the Walker2d task, our method does not outperform the flat RL. Our hypothesis is that the agent is more unstable in this task. In our current setup, the decoder D does not depend on the state S. It acts like an open-loop controller that is more vulnerable to unstable and fast-changing states. However, our option model can be extended to close-loop sub-policies π(a|o, s) that also depends on the state. We will explore this idea in the future study. The different termination conditions have pro and cons. The Fix-len termination condition is more simple and stable in most of the cases. The ones with Term-output and Term-predict termination condition get better in some cases, but they require more tuning of the parameters such as the predict threshold, which controls the average length of the sequences. The experiences used for training the option model is a critical factor that decides what the agent can learn (as humans can also be biased by their previous experiences). We tested two ways of selecting the experiences: random selecting the experiences and selecting experiences with higher rewards. From our experiments, the agent learns faster and performs better in the latter setting. The TAIC framework provides an efficient way to transfer the past experiences to unseen and more sophisticated tasks. We apply the options learned from the above MoJoCo Ant-v1 running task, to novel tasks shown in Figure 4 (c). In the Ant turn task, the agent is awarded by making left turn as fast as possible. The goal of the other three tasks (Ant maze, Ant push and Ant fall) is to move from the start location (orange circle) to the goal location (green circle). The reward is the negative of the distance between the agent and the goal. The last two tasks require the agent to manipulate the red box in a right way, in order to avoid being stuck in sub-optimal trap states. As shown in Figure 8, TAIC (green) achieves higher rewards and quickly converges to the goal state in the first three tasks. In contrast, the flat RL (blue) method is not able to solve these tasks in 3 We compare the proposed TAIC framework with two baseline methods on more challenging tasks. The TAIC framework efficiently transfers the options learned in simpler Ant-v1 task to these novel and more complex tasks. million interactions without any task transfer treatment. We also apply transfer learning on the flat RL (orange) by using the Ant-v1 policy as an initialization. As expected, transfer learning does not bring comparable benefit to the flat RL, indicating sharing weights between different tasks is less efficient. The last two experiments also show that the transferred policy (orange) may even degrade the performance. This does not happen to our TAIC framework since it transfers high level abstracted knowledge in the form of options. This paper presented a general HRL framework TAIC for learning temporal abstraction from action sequences. We formulate the temporal abstraction problem as learning latent representations (called options) over action sequences. In order to learn a better representation, we derive theoretically on how to regularize the option space and give an applicable solution of adding constraints to option space. In the experiments, we try to reveal the underlying structure of the option space by visualizing the correlation between options and state changes. We showed qualitatively and quantitatively that our options encode meaningful information and benefit the RL training. Furthermore, the TAIC framework provides an efficient tool to transfer the knowledge learned from one task to another. Our framework can be applied together with all kinds of RL optimization algorithms, and can be applied to both discrete and continuous problems. This work brings many new directions for future studies. As we currently learn the RL task and the option separately, the option could not be improved with the improvement of the policy. In theory, it is entirely feasible to jointly optimize the two parts, or at least train them alternately. As mentioned above, the current sub-policy acts like an open-loop controller. So learning a close-loop sub-policy beyond the RNN decoder will be one of the focus areas of our future studies. We would also like to apply the TAIC framework to discrete problems and with other RL algorithms such as DQN and SAC. This could bring more insights to further improve the framework.

Represent the short summary for finding the corresponding scientific article (long)

We propose a novel HRL framework, in which we formulate the temporal abstraction problem as learning a latent representation  of  action  sequence.

represent the tldr for finding the corresponding scientific article.
------
Examples:

"We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance." == "Despite their ability to memorize large datasets, deep neural networks often achieve good generalization performance. However, the differences between the learned solutions of networks which generalize and those which do not remain unclear. Additionally, the tuning properties of single directions (defined as the activation of a single unit or some linear combination of units in response to some input) have been highlighted, but their importance has not been evaluated. Here, we connect these lines of inquiry to demonstrate that a network’s reliance on single directions is a good predictor of its generalization performance, across networks trained on datasets with different fractions of corrupted labels, across ensembles of networks trained on datasets with unmodified labels, across different hyper- parameters, and over the course of training. While dropout only regularizes this quantity up to a point, batch normalization implicitly discourages single direction reliance, in part by decreasing the class selectivity of individual units. Finally, we find that class selectivity is a poor predictor of task importance, suggesting not only that networks which generalize well minimize their dependence on individual units by reducing their selectivity, but also that individually selective units may not be necessary for strong network performance. Recent work has demonstrated that deep neural networks (DNNs) are capable of memorizing extremely large datasets such as ImageNet BID39. Despite this capability, DNNs in practice achieve low generalization error on tasks ranging from image classification BID17 to language translation BID37. These observations raise a key question: why do some networks generalize while others do not?Answers to these questions have taken a variety of forms. A variety of studies have related generalization performance to the flatness of minima and PAC-Bayes bounds BID18 BID20 BID27 BID14, though recent work has demonstrated that sharp minima can also generalize BID13. Others have focused on the information content stored in network weights BID0, while still others have demonstrated that stochastic gradient descent itself encourages generalization BID8 BID34 BID36.Here, we use ablation analyses to measure the reliance of trained networks on single directions. We define a single direction in activation space as the activation of a single unit or feature map or some linear combination of units in response to some input. We find that networks which memorize the training set are substantially more dependent on single directions than those which do not, and that this difference is preserved even across sets of networks with identical topology trained on identical data, but with different generalization performance. Moreover, we found that as networks begin to overfit, they become more reliant on single directions, suggesting that this metric could be used as a signal for early stopping. We also show that networks trained with batch normalization are more robust to cumulative ablations than networks trained without batch normalization and that batch normalization decreases the class selectivity of individual feature maps, suggesting an alternative mechanism by which batch normalization may encourage good generalization performance. Finally, we show that, despite the focus on selective single units in the analysis of DNNs (and in neuroscience; BID21 BID40 BID28 BID9, the class selectivity of single units is a poor predictor of their importance to the network's output. In this study, we will use a set of perturbation analyses to examine the relationship between a network's generalization performance and its reliance upon single directions in activation space. We will then use a neuroscience-inspired measure of class selectivity to compare the selectivity of individual directions across networks with variable generalization performance and examine the relationship between class selectivity and importance. We analyzed three models: a 2-hidden layer MLP trained on MNIST, an 11-layer convolutional network trained on CIFAR-10, and a 50-layer residual network trained on ImageNet. In all experiments, ReLU nonlinearities were applied to all layers but the output. Unless otherwise noted, batch normalization was used for all convolutional networks BID19 . For the ImageNet ResNet, top-5 accuracy was used in all cases. Partially corrupted labels As in BID39, we used datasets with differing fractions of randomized labels to ensure varying degrees of memorization. To create these datasets, a given fraction of labels was randomly shuffled and assigned to images, such that the distribution of labels was maintained, but any true patterns were broken. Ablations We measured the importance of a single direction to the network's computation by asking how the network's performance degrades once the influence of that direction was removed. To remove a coordinate-aligned single direction, we clamped the activity of that direction to a fixed value (i.e., ablating the direction). Ablations were performed either on single units in MLPs or an entire feature map in convolutional networks. For brevity, we will refer to both of these as'units.' Critically, all ablations were performed in activation space, rather than weight space. More generally, to evaluate a network's reliance upon sets of single directions, we asked how the network's performance degrades as the influence of increasing subsets of single directions was removed by clamping them to a fixed value (analogous to removing increasingly large subspaces within activation space). This analysis generates curves of accuracy as a function of the number of directions ablated: the more reliant a network is on low-dimensional activation subspaces, the more quickly the accuracy will drop as single directions are ablated. Interestingly, we found that clamping the activation of a unit to the empirical mean activation across the training or testing set was more damaging to the network's performance than clamping the activation to zero (see Appendix A.1). We therefore clamped activity to zero for all ablation experiments. Addition of noise As the above analyses perturb units individually, they only measure the influence of coordinate-aligned single directions. To test networks' reliance upon random single directions, we added Gaussian noise to all units with zero mean and progressively increasing variance. To scale the variance appropriately for each unit, the variance of the noise added was normalized by the empirical variance of the unit's activations across the training set. To quantify the class selectivity of individual units, we used a metric inspired by the selectivity indices commonly used in systems neuroscience BID12 BID9 Figure 1: Memorizing networks are more sensitive to cumulative ablations. Networks were trained on MNIST (2-hidden layer MLP, a), CIFAR-10 (11-layer convolutional network, b), and ImageNet (50-layer ResNet, c). In a, all units in all layers were ablated, while in b and c, only feature maps in the last three layers were ablated. Error bars represent standard deviation across 10 random orderings of units to ablate. & ). The class-conditional mean activity was first calculated across the test set, and the selectivity index was then calculated as follows: DISPLAYFORM0 with µ max representing the highest class-conditional mean activity and µ −max representing the mean activity across all other classes. For convolutional feature maps, activity was first averaged across all elements of the feature map. This metric varies from 0 to 1, with 0 meaning that a unit's average activity was identical for all classes, and 1 meaning that a unit was only active for inputs of a single class. We note that this metric is not a perfect measure of information content in single units; for example, a unit with a little information about every class would have a low class selectivity index. However, it does measure the discriminability of classes along a given direction. The selectivity index also identifies units with the same class tuning properties which have been highlighted in the analysis of DNNs BID21 BID38 BID11 BID40 BID28. However, in addition to class selectivity, we replicate all of our using mutual information, which, in contrast to class selectivity, should highlight units with information about multiple classes, and we find qualitively similar outcomes (Appendix A.5). We also note that while a class can be viewed as a highly abstract feature, implying that our may generalize to feature selectivity, we do not examine feature selectivity in this work. Here, we provide a rough intuition for why a network's reliance upon single directions might be related to generalization performance. Consider two networks trained on a large, labeled dataset with some underlying structure. One of the networks simply memorizes the labels for each input example and will, by definition, generalize poorly ('memorizing network') while the other learns the structure present in the data and generalizes well ('structure-finding network'). The minimal description length of the model should be larger for the memorizing network than for the structurefinding network. As a , the memorizing network should use more of its capacity than the structure-finding network, and by extension, more single directions. Therefore, if a random single direction is perturbed, the probability that this perturbation will interfere with the representation of the data should be higher for the memorizing network than for the structure-finding network 2. To test whether memorization leads to greater reliance on single directions, we trained a variety of network types on datasets with differing fractions of randomized labels and evaluated their performance as progressively larger fractions of units were ablated (see Sections 2.2 and 2.1). By definition, these curves must begin at the network's training accuracy (approximately 1 for all networks tested) and fall to chance levels when all directions have been ablated. To rule out variance due to the specific order of unit ablation, all experiments were performed with mutliple random ablation orderings of units. As many of the models were trained on datasets with corrupted labels and, by definition, cannot generalize, training accuracy was used to evaluate model performance. Consistent with our intuition, we found that networks trained on varying fractions of corrupted labels were significantly more sensitive to cumulative ablations than those trained on datasets comprised of true labels, though curves were not always perfectly ordered by the fraction of corrupted labels (Fig. 1).We next asked whether this effect was present if networks were perturbed along random bases. To test this, we added noise to each unit (see Section 2.2). Again, we found that networks trained on corrupted labels were substantially and consistently more sensitive to noise added along random bases than those trained on true labels (Fig. 2).The above apply to networks which are forced to memorize at least a portion of the training set -there is no other way to solve the task. However, it is unclear whether these would apply to networks trained on uncorrupted data. In other words, do the solutions found by networks with the same topology and data, but different generalization performance exhibit differing reliance upon single directions? To test this, we trained 200 networks on CIFAR-10, and evaluated their generalization error and reliance on single directions. All networks had the same topology and were trained on the same dataset (unmodified CIFAR-10). Individual networks only differed in their random initialization (drawn from identical distributions), the data order used during training, and their learning rate 3. We found that the 5 networks with the best generalization performance were more robust to the ablation of single directions than the 5 networks with the worst generalization performance (Fig. 3a). To quantify this further, we measured the area under the ablation curve for each of the 200 networks and plotted it as a function of generalization error (Fig. 3b). Interestingly, networks appeared to undergo a discrete regime shift in their reliance upon single directions; however, this effect might have been caused by degeneracy in the set of solutions found by the optimization procedure, and we note that there was also a negative correlation present within clusters (e.g., top left cluster). These demonstrate that the relationship between generalization performance and single direction reliance is not merely a side-effect of training with corrupted labels, but is instead present even among sets networks with identical training data. This relationship raises an intriguing question: can single direction reliance be used to estimate generalization performance without the need for a held-out test set? And if so, might it be used as a signal for early stopping or hyperpameter selection? As a proof-of-principle experiment for early stopping, we trained an MLP on MNIST and measured the area under the cumulative ablation curve (AUC) over the course of training along with the train and test loss. Interestingly, we found that the point in training at which the AUC began to drop was the same point that the train and test loss started to diverge (Fig. 4a). Furthermore, we found that AUC and test loss were negatively correlated (Spearman's correlation: -0.728; Fig. 4b).As a proof-of-principle experiment for hyperparameter selection, we trained 192 CIFAR-10 models with different hyperparemeter settings (96 hyperparameters with 2 repeats each; see Appendix A.2). We found that AUC and test accuracy were highly correlated (Spearman's correlation: 0.914; Fig. 4c), and by performing random subselections of 48 hyperparameter settings, AUC selected one of the top 1, 5, and 10 settings 13%, 83%, and 98% of the time, respectively, with an average difference in test accuracy between the best model selected by AUC and the optimal model of only 1 ± 1.1% (mean ± std). These suggest that single direction reliance may serve as a good proxy for hyperparameter selection and early stopping, but further work will be necessary to evaluate whether these hold in more complicated datasets. Dropout Our experiments are reminiscent of using dropout at training time, and upon first inspection, dropout may appear to discourage networks' reliance on single directions BID35. However, while dropout encourages networks to be robust to cumulative ablations up until the dropout fraction used in training, it should not discourage reliance on single directions past that point. Given enough capacity, a memorizing network could effectively guard against dropout by merely copying the information stored in a given direction to several other directions. However, the network will only be encouraged to make the minimum number of copies necessary to guard against the dropout fraction used in training, and no more. In such a case, the network would be robust to dropout so long as all redundant directions were not simultaneously removed, yet still be highly reliant on single directions past the dropout fraction used in training. To test whether this intuition holds, we trained MLPs on MNIST with dropout probabilities ∈ {0.1, 0.2, 0.3} on both corrupted and unmodified labels. Consistent with the observation in BID4, we found that networks with dropout trained on randomized labels required more epochs to converge and converged to worse solutions at higher dropout probabilities, suggesting that dropout does indeed discourage memorization. However, while networks trained on both corrupted and unmodified labels exhibited minimal loss in training accuracy as single directions were removed up to the dropout fraction used in training, past this point, networks trained on randomized labels were much more sensitive to cumulative ablations than those trained on unmodified labels (Fig. 5a). Interestingly, networks trained on unmodified labels with different dropout fractions were all similarly robust to cumulative ablations. These suggest that while dropout may serve as an effective regularizer to prevent memorization of randomized labels, it does not prevent over-reliance on single directions past the dropout fraction used in training. Batch normalization In contrast to dropout, batch normalization does appear to discourage reliance upon single directions. To test this, we trained convolutional networks on CIFAR-10 with and without batch normalization and measured their robustness to cumulative ablation of single directions. Networks trained with batch normalization were consistently and substantially more robust to these ablations than those trained without batch normalization (Fig. 5b). This suggests that in a bFigure 6: Batch normalization decreases class selectivity and increases mutual information. Distributions of class selectivity (a) and mutual information (b) for networks trained with (blue) and without batch normalization (purple). Each distribution comprises 4 model instances trained on uncorrupted CIFAR-10.addition to reducing covariate shift, as has been proposed previously BID19, batch normalization also implicitly discourages reliance upon single directions. Our thus far suggest that networks which are less reliant on single directions exhibit better generalization performance. This may appear counter-intuitive in light of extensive past work in both neuroscience and deep learning which highlights single units or feature maps which are selective for particular features or classes BID21 BID38 BID11 BID40 BID28. Here, we will test whether the class selectivity of single directions is related to the importance of these directions to the network's output. First, we asked whether batch normalization, which we found to discourage reliance on single directions, also influences the distribution of information about class across single directions. We used the selectivity index described above (see Section 2.3) to quantify the discriminability between classes based on the activations of single feature maps across networks trained with and without batch normalization. Interestingly, we found that while networks trained without batch normalization exhibited a large fraction of feature maps with high class selectivity 4, the class selectivity of feature maps in networks trained with batch normalization was substantially lower (Fig. 6a). In contrast, we found that batch normalization increases the mutual information present in feature maps (Fig. 6b). These suggest that batch normalization actually discourages the presence of feature maps with concentrated class information and rather encourages the presence of feature maps with information about multiple classes, raising the question of whether or not such highly selective feature maps are actually beneficial. We next asked whether the class selectivity of a given unit was predictive of the impact on the network's loss of ablating said unit. Since these experiments were performed on networks trained on unmodified labels, test loss was used to measure network impact. For MLPs trained on MNIST, we found that there was a slight, but minor correlation (Spearman's correlation: 0.095) between a unit's class selectivity and the impact of its ablation, and that many highly selective units had minimal impact when ablated FIG2. By analyzing convolutional networks trained on CIFAR-10 and ImageNet, we again found that, across layers, the ablation of highly selective feature maps was no more impactful than the ablation of non-selective feature maps FIG2. In fact, in the CIFAR-10 networks, there was actually a negative correlation between class selectivity and feature map importance (Spearman's correlation: -0.428, FIG2). To test whether this relationship was depth-dependent, we calculated the correlation between class selectivity and importance separately for each layer, and found that the vast majority of the negative correlation was driven by early layers, while later layers exhibited no relationship between class selectivity and importance FIG2. Interestingly, in all three networks, ablations in early layers were more impactful than ablations in later layers, consistent with theoretical observations BID29. Additionally, we performed all of the above experiments with mutual information in place of class selectivity, and found qualitatively similar (Appendix A.5). As a final test, we compared the class selectivity to the L 1 -norm of the filter weights, a metric which has been found to be a successful predictor of feature map importance in the model pruning literature BID22. Consistent with our previous observations, we found that class selectivity was largely unrelated to the L 1 -norm of the filter weights, and if anything, the two were negatively correlated (Fig. A3, see Appendix A.4 for details). Taken together, these suggest that class selectivity is not a good predictor of importance, and imply that class selectivity may actually be detrimental to network performance. Further work will be necessary to examine whether class and/or feature selectivity is harmful or helpful to network performance. Much of this work was directly inspired by BID39, and we replicate their using partially corrupted labels on CIFAR-10 and ImageNet. By demonstrating that memorizing networks are more reliant on single directions, we also provide an answer to one of the questions they posed: is there an empirical difference between networks which memorize and those which generalize?Our work is also related to work linking generalization and the sharpness of minima BID18 BID20 BID27. These studies argue that flat minima generalize better than sharp minima (though BID13 recently found that sharp minima can also generalize well). This is consistent with our work, as flat minima should correspond to solutions in which perturbations along single directions have little impact on the network output. Another approach to generalization has been to contextualize it in information theory. For example, BID0 demonstrated that networks trained on randomized labels store more information in their weights than those trained on unmodfied labels. This notion is also related to BID33, which argues that during training, networks proceed first through a loss minimization phase followed by a compression phase. Here again, our work is consistent, as networks with more information stored in their weights (i.e., less compressed networks) should be more reliant upon single directions than compressed networks. More recently, BID4 analyzed a variety of properties of networks trained on partially corrupted labels, relating performance and time-to-convergence to capacity. They also demonstrated that dropout, when properly tuned, can serve as an effective regularizer to prevent memorization. However, we found that while dropout may discourage memorization, it does not discourage reliance on single directions past the dropout probability. We found that class selectivity is a poor predictor of unit importance. This observation is consistent with a variety of recent studies in neuroscience. In one line of work, the benefits of neural systems which are robust to coordinate-aligned noise have been explored BID6 Perturbation analyses have been performed for a variety of purposes. In the model pruning literature, many studies have removed units with the goal of generating smaller models with similar performance BID22 BID3 BID24, and recent work has explored methods for discovering maximally important directions BID30 ). Recently, BID10 used cumulative ablations to measure network robustness, though the relationship to generalization was not explored. A variety of studies within deep learning have highlighted single units which are selective for features or classes BID21 BID38 BID11 BID40 BID28 BID1. Additionally, BID1 analyzed the minimum number of sufficient feature maps (sorted by a measure of selectivity) to achieve a given accuracy. However, none of the above studies has tested the relationship between a unit's class selectivity or information content and its necessity to the network's output. BID7 have quantified a related metric, concept selectivity, across layers and networks, finding that units get more concept-selective with depth, which is consistent with our own observations regarding class selectivity (see Appendix A.3). However, they also observed a correlation between the number of concept-selective units and performance on the action40 dataset across networks and architectures. It is difficult to compare these directly, as the data used are substantially different as is the method of evaluating selectivity. Nevertheless, we note that BID7 measured the absolute number of concept-selective units across networks with different total numbers of units and depths. The relationship between the number of concept-selective units and network performance may therefore arise as a of a larger number of total units (if a fixed fraction of units is concept-selective) and increased depth (we both observed that selectivity increases with depth). In this work, we have taken an empirical approach to understand what differentiates neural networks which generalize from those which do not. Our experiments demonstrate that generalization capability is related to a network's reliance on single directions, both in networks trained on corrupted and uncorrupted data, and over the course of training for a single network. They also show that batch normalization, a highly successful regularizer, seems to implicitly discourage reliance on single directions. One clear extension of this work is to use these observations to construct a regularizer which more directly penalizes reliance on single directions. As it happens, the most obvious candidate to regularize single direction reliance is dropout (or its variants), which, as we have shown, does not appear to regularize for single direction reliance past the dropout fraction used in training (Section 3.3). Interestingly, these suggest that one is able to predict a network's generalization performance without inspecting a held-out validation or test set. This observation could be used in several interesting ways. First, in situations where labeled training data is sparse, testing networks' reliance on single directions may provide a mechanism to assess generalization performance without sacrificing training data to be used as a validation set. Second, by using computationally cheap empirical measures of single direction reliance, such as evaluating performance at a single ablation point or sparsely sampling the ablation curve, this metric could be used as a signal for early-stopping or hyperparameter selection. We have shown that this metric is viable in simple datasets (Section 3.2), but further work will be necessary to evaluate viability in more complicated datasets. Another interesting direction for further research would be to evaluate the relationship between single direction reliance and generalization performance across different generalization regimes. In this work, we evaluate generalization in which train and test data are drawn from the same distribution, but a more stringent form of generalization is one in which the test set is drawn from a unique, but overlapping distribution with the train set. The extent to which single direction reliance depends on the overlap between the train and test distributions is also worth exploring in future research. This work makes a potentially surprising observation about the role of individually selective units in DNNs. We found not only that the class selectivity of single directions is largely uncorrelated with their ultimate importance to the network's output, but also that batch normalization decreases the class selectivity of individual feature maps. This suggests that highly class selective units may actually be harmful to network performance. In addition, it implies than methods for understanding neural networks based on analyzing highly selective single units, or finding optimal inputs for single units, such as activation maximization BID15 ) may be misleading. Importantly, as we have not measured feature selectivity, it is unclear whether these will generalize to featureselective directions. Further work will be necessary to clarify all of these points. To remove the influence of a given direction, its value should be fixed or otherwise modified such that it is no longer dependent on the input. However, the choice of such a fixed value can have a substantial impact. For example, if its value were clamped to one which is highly unlikely given its distribution of activations across the training set, network performance would likely suffer drastically. Here, we compare two methods for ablating directions: ablating to zero and ablating to the empirical mean over the training set. Using convolutional networks trained on CIFAR-10, we performed cumulative ablations, either ablating to zero or to the feature map's mean (means were calculated independently for each element of the feature map), and found that ablations to zero were significantly less damaging than ablations to the feature map's mean FIG4. Interestingly, this corresponds to the ablation strategies generally used in the model pruning literature BID22 BID3 BID24. MNIST MLPs For class selectivity, generalization, early stopping, and dropout experiments, each layer contained 128, 512, 2048 and 2048 units, respectively. All networks were trained for 640 epochs, with the exception of dropout networks which were trained for 5000 epochs. CIFAR-10 ConvNets Convolutional networks were all trained on CIFAR-10 for 100 epochs. Layer sizes were: 64, 64, 128, 128, 128, 256, 256, 256, 512, 512, 512, with strides of 1, 1, 2, 1, 1, 2, 1, 1, 2, 1, 1, respectively. All kernels were 3x3. For the hyperparameter sweep used in Section 3.2, learning rate and batch size were evaluated using a grid search. ImageNet ResNet 50-layer residual networks BID17 were trained on ImageNet using distributed training with 32 workers and a batch size of 32 for 200,000 steps. Blocks were structured as follows (stride, filter sizes, output channels): (1x1, 64, 64, 256) x 2, (2x2, 64, 64, 256), (1x1, 128, 128, 512) x 3, (2x2, 128, 128, 512), (1x1, 256, 256, 1024) x 5, (2x2, 256, 256, 1024), (1x1, 512, 512, 2048) x 3. For training with partially corrupted labels, we did not use any data augmen- tation, as it would have dramatically increasing the effective training set size, and hence prevented memorization. Here, we evaluate the distribution of class selectivity as a function of depth. In both networks trained on CIFAR-10 (Fig. A2a) and ImageNet (Fig. A2b), selectivity increased as a function of depth. This is consistent with BID7, who show that concept-selectivity increases with depth. It is also consistent with BID2, who show depth increases the linear decodability of class information (though they evaluate linear decodability based on an entire layer rather than a single unit).A.4 RELATIONSHIP BETWEEN CLASS SELECTIVITY AND THE FILTER WEIGHT L 1 -NORM Importantly, our on the lack of relationship between class selectivity and importance do not suggest that there are not directions which are more or less important to the network's output, nor do they suggest that these directions are not predictable; they merely suggest that class selectivity is not a good predictor of importance. As a final test of this, we compared class selectivity to the L 1 -norm of the filter weights, a metric which has been found to be a strongly correlated with the impact of removing a filter in the model pruning literature BID22. Since the L 1 -norm of the filter weights is predictive of impact of a feature map's removal, if class selectivity is also a good predictor, the two metrics should be correlated. In the ImageNet network, we found that there was no correlation between the L 1 -norm of the filter weights and the class selectivity (Fig. A3a), while in the CIFAR-10 network, we found there was actually a negative correlation (Fig. A3b). To examine whether mutual information, which, in contrast to class selectivity, highlights units with information about multiple classes, is a good predictor of importance, we performed the same experiments as in Section 3.4 with mutual information in place of class selectivity. We found, that while the were a little less consistent (e.g., there appears to be some relationship in very early and very late layers in CIFAR-10), mutual information was generally a poor predictor of unit importance FIG6."

DCEM learns latent domains for optimization problems and helps bridge the gap between model-based and model-free RL --- we create a differentiable controller and fine-tune parts of it with PPO


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Recurrent neural networks (RNNs) have achieved state-of-the-art performance on many diverse tasks, from machine translation to surgical activity recognition, yet training RNNs to capture long-term dependencies remains difficult. To date, the vast majority of successful RNN architectures alleviate this problem using nearly-additive connections between states, as introduced by long short-term memory (LSTM). We take an orthogonal approach and introduce MIST RNNs, a NARX RNN architecture that allows direct connections from the very distant past. We show that MIST RNNs 1) exhibit superior vanishing-gradient properties in comparison to LSTM and previously-proposed NARX RNNs; 2) are far more efficient than previously-proposed NARX RNN architectures, requiring even fewer computations than LSTM; and 3) improve performance substantially over LSTM and Clockwork RNNs on tasks requiring very long-term dependencies. Recurrent neural networks BID33; BID35 ) are a powerful class of neural networks that are naturally suited to modeling sequential data. For example, in recent years alone, RNNs have achieved state-of-the-art performance on tasks as diverse as machine translation, speech recognition BID29, generative image modeling BID30, and surgical activity recognition BID8.These successes, and the vast majority of other RNN successes, rely on a mechanism introduced by long short-term memory BID20 BID14, which was designed to alleviate the so called vanishing gradient problem (; BID3 . The problem is that gradient contributions from events at time t − τ to a loss at time t diminish exponentially fast with τ, thus making it extremely difficult to learn from distant events (see FIG0 . LSTM alleviates the problem using nearly-additive connections between adjacent states, which help push the base of the exponential decay toward 1. However LSTM in no way solves the problem, and in many cases still fails to learn long-term dependencies (see, e.g., BID0). 1 RNNs BID27 offer an orthogonal mechanism for dealing with the vanishing gradient problem, by allowing direct connections, or delays, from the distant past. However NARX RNNs have received much less attention in literature than LSTM, which we believe is for two reasons. First, as previously introduced, NARX RNNs have only a small effect on vanishing gradients, as they reduce the exponent of the decay by only a factor of n d, the number of delays. Second, as previously introduced, NARX RNNs are extremely inefficient, as both parameter counts and computation counts grow by the same factor n d.In this paper, we introduce MIxed hiSTory RNNs (MIST RNNs), a new NARX RNN architecture which 1) exhibits superior vanishing-gradient properties in comparison to LSTM and previouslyproposed NARX RNNs; 2) improves performance substantially over LSTM on tasks requiring very long-term dependencies; and 3) remains efficient in parameters and computation, requiring even fewer than LSTM for a fixed number of hidden units. Importantly, MIST RNNs reduce the decay's exponent by a factor of 2 n d −1; see FIG1. 2 AND RELATED WORK Recurrent neural networks, as commonly described in literature, take on the general form DISPLAYFORM0 which compute a new state h t in terms of the previous state h t−1, the current input x t, and some parameters θ (which are shared over time).One of the earliest variants, now known to be especially vulnerable to the vanishing gradient problem, is that of simple RNNs , described by DISPLAYFORM1 In this equation and elsewhere in this paper, all weight matrices W and biases b collectively form the parameters θ to be learned, and tanh is always written explicitly 2.Long short-term memory BID20 BID14, the most widelyused RNN architecture to date, was specifically introduced to address the vanishing gradient problem. The term LSTM is often overloaded; we refer to the variant with forget gates and without peephole connections, which performs similarly to more complex variants BID16: DISPLAYFORM2 Here σ(·) denotes the element-wise sigmoid function and denotes element-wise multiplication. f t, i t, and o t are referred as the forget, input, and output gates, which can be interpreted as controlling how much we reset, write to, and read from the memory cell c t. LSTM has better gradient properties than simple RNNs (see FIG1) because of the mechanism in Equation 7, which introduces a path between c t−1 and c t which is modulated only by the forget gate. We also remark that gated recurrent units BID5 alleviate the vanishing gradient problem using this exact same idea. NARX RNNs BID27 also address the vanishing gradient problem, but using a mechanism that is orthogonal to (and possibly complementary to) that of LSTM. This is done by allowing delays, or direct connections from the past. NARX RNNs in their general form are described by but literature typically assumes the specific variant explored in BID27, DISPLAYFORM3 DISPLAYFORM4 which we refer to as simple NARX RNNs. Note that simple NARX RNNs require approximately n d as much computation and n d as many parameters as their simple-RNN counterpart (with n d = 1), which greatly hinders their applicability in practice. To our knowledge, this drawback holds for all NARX RNN variants before MIST RNNs. For example, in , higher-order recurrent neural networks (HORNNs) are defined precisely as simple NARX RNNs, and every variant in the paper suffers from this exact same problem. And, in BID37, a simple NARX RNN architecture is defined that is limited to having precisely two delays with non-zero weights. This way, at the expense of having fewer, longer paths to the past, parameter and computation counts are only doubled. The previous work that is most similar to ours is that of Clockwork RNNs BID22, which split weights and hidden units into partitions, each with a distinct period. When it's not a partition's time to tick, its hidden units are passed through unchanged, and so Clockwork RNNs in some ways mimic NARX RNNs. However Clockwork RNNs differ in two key ways. First, Clockwork RNNs sever high-frequency-to-low-frequency paths, thus making it difficult to learn long-term behavior that must be detected at high frequency (for example, learning to depend on quick motions from the past for activity recognition). Second, Clockwork RNNs require hidden units to be partitioned a priori, which in practice is difficult to do in any meaningful way. NARX RNNs (and in particular MIST RNNs) suffer from neither of these drawbacks. Many other approaches have also been proposed to capture long-term dependencies. Notable approaches include maintaining a generative model over inputs and learning to process only unexpected inputs , operating explicitly at multiple time scales BID9, Hessian-free optimization BID28, using associative or explicit memory BID32 BID7 BID15 BID34, and initializing or restricting weight matrices to be orthogonal BID0 BID18. In BID3 BID31, gradient decompositions and sufficient conditions for vanishing gradients are presented for simple RNNs, which contain one path between times t − τ and t. Here, we use the chain rule for ordered derivatives to connect gradient components to paths and edges, which in turn provides a simple extension of the from BID3 BID31 to general NARX RNNs. We remark that we rely on slightly overloaded notation for clarity, as otherwise notation becomes cumbersome (see ).We begin by disambiguating notation, as the symbol ∂f ∂x is routinely overloaded in literature. Consider the Jacobian of f (x, u(x)) with respect to x. We let DISPLAYFORM0 ∂x, a collection of full derivatives, and we let DISPLAYFORM1 ∂x, a collection of partial derivatives. This lets us write the ordinary chain rule as DISPLAYFORM2 ∂x. Note that this notation is consistent with (; BID10, but is the exact opposite of the convention used in BID31 . Consider an ordered system of n vectors v 1, v 2, . . ., v n, where each is a function of all previous: DISPLAYFORM0 The chain rule for ordered derivatives expresses the full derivatives DISPLAYFORM1 ∂vj for any j < i in terms of the full derivatives that relate v i to all previous v k : DISPLAYFORM2 Consider NARX RNNs in their general form FORMULA3), which we remark encompasses other RNNs such as LSTM as special cases. Also, for simplicity, consider the situation that is most often encountered in practice, where the loss at time t is defined in terms of the current state h t and its own parameters θ l (which are independent of θ). DISPLAYFORM0 (This is in not necessary, but we proceed this way to make the connection with RNNs in practice evident. For example, f l may be a linear transformation with parameters θ l followed by squarederror loss.) Then the Jacobian (or transposed gradient) with respect to θ can be written as DISPLAYFORM1 because the additional term DISPLAYFORM2, and so on in Equations 11 and 12, we immediately obtain DISPLAYFORM3 because all of the partials ∂xt−τ ∂θ are 0. Equations 14 and 15 extend Equations 3 and 4 of BID31 to general NARX RNNs, which encompass simple RNNs, LSTM, etc., as special cases. This decomposition breaks ∂ + ht ∂θ into its temporal components, making it clear that the spectral norm of DISPLAYFORM4 ∂ht−τ plays a major role in how h t−τ affects the final gradient DISPLAYFORM5 In particular, if the norm of DISPLAYFORM6 ∂ht−τ is extremely small, then h t−τ has only a negligible effect on the final gradient, which in turn makes it extremely difficult to learn from events that occurred at t − τ. Equations 14 and 15, along with the chain rule for ordered derivatives, let us connect gradient components to paths and edges, which is useful for a) gaining insights into various architectures and b) solidifying intuitions from backpropagation through time which suggest that short paths between t − τ and t facilitate gradient flow. Here we provide an overview of the main idea; please see the appendix for a full derivation. By applying the chain rule for ordered derivatives to expand DISPLAYFORM0, we obtain a sum over τ terms. However each term involves a partial derivative between h t and a prior hidden state, and thus all of these terms are 0 with the exception of those states that share an edge with h t. Now, for each term, we can repeat this process. This then yields non-zero terms only for hidden states which can be connected to h t through two edges. We can then continue to apply the chain rule for ordered derivatives repeatedly, until only partial derivatives remain. Upon completion, we have a sum over gradient components, with each component corresponding to exactly one path from t − τ to t and being a product over its path's edges. The spectral norm corresponding to any particular path (t − τ → t → t → · · · → t) can then bounded as DISPLAYFORM1 where λ is the maximum spectral norm of any factor and n e is the number of edges on the path. Terms with λ < 1 diminish exponentially fast, and when all λ < 1, shortest paths dominate 3. Viewing gradient components as paths, with each component being a product with one factor per edge along the path, gives us useful insight into various RNN architectures. When relating a loss at time t to events at time t − τ, simple RNNs and LSTM contain shortest paths of length τ, while simple NARX RNNs contain shortest paths of length τ /n d, where n d is the number of delays. One can envision many NARX RNN architectures with non-contiguous delays that reduce these shortest paths further. In this section we introduce one such architecture using base-2 exponential delays. In this case, for all τ ≤ 2 n d −1, shortest paths exist with only log 2 τ edges; and for τ > 2 n d −1, shortest paths exist with only τ /2 n d −1 edges (see FIG0). Finally we must avoid the parameter and computation growth of simple NARX RNNs. We achieve this by sharing weights over delays, instead using an attention-like mechanism BID2 over delays and a reset mechanism from gated recurrent units BID5 ).The proposed architecture, which we call mixed history RNNs (MIST RNNs), is described by DISPLAYFORM0 DISPLAYFORM1 Here, a t is a learned vector of n d convex-combination coefficients and r t is a reset gate. At each time step, a convex combination of delayed states is formed according to a t; units of this combination are reset according to r t; and finally the typical linear layer and nonlinearity are applied. Here we compare MIST RNNs to simple RNNs, LSTM, and Clockwork RNNs. We begin with the sequential permuted MNIST task and the copy problem, synthetic tasks that were introduced to explicitly test RNNs for their ability to learn long-term dependencies BID20 BID28 BID24 BID0 BID18 BID7. Next we move on to 3 tasks for which it is plausible that very long-term dependencies play a role: recognizing surgical maneuvers from robot kinematics, recognizing phonemes from speech, and classifying activities from smartphone motion data. We note that for all architectures involved, many variations can be applied (variational dropout, layer normalization, zoneout, etc.). We keep experiments manageable by comparing architectures without such variations. The sequential MNIST task BID24 consists of classifying 28x28 MNIST images BID25 as one of 10 digits, by scanning pixel by pixel -left to right, top to bottom -and emitting FORMULA0 is a challenging variant where a random permutation of pixels is chosen and applied to all images before classification. LSTM with 100 hidden units is used as a baseline, with hidden unit counts for other architectures chosen to match the number of parameters. Means and standard deviations are computed using the top 5 randomized trials out of 50 (ranked according to performance on the validation set), with random learning rates and initializations. Additional experimental details can be found in the appendix. Test error rates are shown in TAB0. Here, MIST RNNs outperform simple RNNs, LSTM, and Clockwork RNNs by a large margin. We remark that our LSTM error rates are consistent with best previously-reported values, such as the error rates of 9.8% in BID6 and 12% in BID0, which also use 100 hidden units. One may also wonder if the difference in performance is due to hidden-unit counts. To test this we also increased the LSTM hidden unit count to 139 (to match MIST RNNs), and continued to increase the capacity of each model further. MIST RNNs significantly outperform LSTM in all cases. We also used this task to visualize gradient magnitudes as a function of τ (the distance from the loss which occurs at time t = 784). Gradient norms for all methods were averaged over a batch of 100 random examples early in training; see FIG1. Here we can see that simple RNNs and LSTM capture essentially no learning signal from steps that are far from the loss. To validate this claim further, we repeated the 512-unit LSTM and MIST RNN experiments, but using only the last 200 permuted pixels (rather than all 784). LSTM performance remains the same (7.4% error, within 1 standard deviation) whereas MIST RNN performance drops by 15 standard deviations (6.0% error). The copy problem is a synthetic task that explicitly challenges a network to store and reproduce information from the past. Our setup follows BID0, which is in turn based on BID20 ). An input sequence begins with L relevant symbols to be copied, is followed by a delay of D − 1 special blank symbols and 1 special go symbol, and ends with L additional blank symbols. The corresponding target sequence begins with L + D blank symbols and ends with a copy of the relevant symbols from the inputs (in the same order). We run experiments with copy delays of D = 50, 100, 200, and 400. LSTM with 100 hidden units is used as a baseline, with hidden unit counts for other architectures chosen to match the number of parameters. Additional experimental details can be found in the appendix. Results are shown in FIG2, showing validation curves of the top 5 randomized trials out of 50, with random learning rates and initializations. With a short copy delay of D = 50, we can see that all methods other than Clockwork RNNs can solve the task in a reasonable amount of time. However, as the copy delay D is increased, we can see that simple RNNs and LSTM become unable to learn a solution, whereas MIST RNNs are relatively unaffected. We also note that our LSTM are consistent with those in BID0 BID18.Note that Clockwork RNNs are expected to fail for large delays (for example, the second symbol can only be seen by the highest-frequency partition, so learning to copy this symbol will fail for precisely the same reason that simple RNNs fail). However, here they also fail for short delays, which is surprising because the high-speed partition resembles a simple RNN. We hypothesized that this failure is due to hidden unit counts / parameter counts: here, the high-frequency partition is allocated only 256 / 8 = 32 hidden units. To test this hypothesis, we reran the Clockwork RNN experiments with 1024 hidden units, so that 128 are allocated to the high-frequency partition. Indeed, under this configuration (with 10x as many parameters), Clockwork RNNs do solve the task for a delay of D = 50 and fail to solve the task for all higher delays, thus behaving like simple RNNs. Here we consider the task of online surgical maneuver recognition using the MISTIC-SL dataset BID12 BID8. Maneuvers are fairly long, high-level activities; examples include suture throw and knot tying. The dataset was collected using a da Vinci, and the goal is to map robot kinematics over time (e.g., x, y, z) to gestures over time (which are densely labeled as 1 of 4 maneuvers on a per-frame basis). We follow BID8, which achieves state-ofthe-art performance on this task, as closely as possible, using the same kinematic inputs, test setup, and hyperparameters; details can be found in the original work or in the appendix. The primary difference is that we replace their LSTM layer with our layers. Results are shown in TAB1. Here MIST RNNs match LSTM performance (with half the number of parameters). Here we consider the task of online framewise phoneme recognition using the TIMIT corpus BID13. Each frame is originally labeled as 1 of 61 phonemes. We follow common practice and collapse these into a smaller set of 39 phonemes BID26, and we include glottal stops to yield 40 classes in total. We follow BID16 for data preprocessing and for training, validation, and test splits. LSTM with 100 hidden units is used as a baseline, with hidden unit counts for other architectures chosen to match the number of parameters. Means and standard deviations are computed using the top 5 randomized trials out of 50 (ranked according to performance on the validation set), with random learning rates and initializations. Other experimental details can be found in the appendix. TAB2 shows that LSTM and MIST RNNs perform nearly identically, which both outperform simple RNNs and Clockwork RNNs. Here we consider the task of sequence classification from smartphones using the MobiAct (v2.0) dataset BID4. The goal is to classify each sequence as jogging, running, sitting down, etc., using smartphone motion data over time. Approximately 3,200 sequences were collected from 67 different subjects. We use the first 47 subjects for training, the next 10 for validation, and the final 10 for testing. Means and standard deviations are computed using the top 5 randomized trials out of 50 (ranked according to performance on the validation set), with random learning rates and initializations. Other experimental details can be found in the appendix. Results are shown in TAB3. Here, MIST RNNs outperform all other methods, including LSTM and LSTM +, a variant with the same number of hidden units and twice as many parameters. In this work we analyzed NARX RNNs and introduced a variant which we call MIST RNNs, which 1) exhibit superior vanishing-gradient properties in comparison to LSTM and previously-proposed NARX RNNs; 2) improve performance substantially over LSTM on tasks requiring very long-term dependencies; and 3) require even fewer parameters and computation than LSTM. One obvious direction for future work is the exploration of other NARX RNN architectures with non-contiguous delays. In addition, many recent techniques that have focused on LSTM are immediately transferable to NARX RNNs, such as variational dropout BID11, layer normalization BID1, and zoneout BID23, and it will be interesting to see if such enhancements can improve MIST RNN performance further. Removed for anonymity. ∂ht−τ are 0 except for the one satisfying t = t − τ + 1. This yields DISPLAYFORM0 Now, by applying Equation 12 again to DISPLAYFORM1, and then to DISPLAYFORM2, and so on, we trace out a path from t − τ to t, as shown in FIG0, finally ing the single term DISPLAYFORM3 which is associated with the only path from t − τ to t, with one factor for each edge that is encountered along the path. Next we consider simple NARX RNNs, again by expanding Equation 15. From Equation 10, we can see that up to n d partials are now nonzero, and that any particular partial ∂h t ∂ht−τ is nonzero if and only if t > t − τ and t and t − τ share an edge. Collecting these t as the set V t−τ = {t : t > t − τ and (t − τ, t) ∈ E}, we can write DISPLAYFORM0 We can then apply this exact same process to each DISPLAYFORM1; by defining V t = {t : t > t and (t, t) ∈ E} for all t, we can write DISPLAYFORM2 By continuing this process until only partials remain, we obtain a summation over all possible paths from t − τ to t. Each term in the sum is a product over factors, one per edge: DISPLAYFORM3 The analysis is nearly identical for general NARX RNNs, with the only difference being the specific sets of edges that are considered.8 APPENDIX: EXPERIMENTAL DETAILS 8.1 GENERAL EXPERIMENTAL SETUP Everything in this section holds for all experiments except surgical maneuver recognition, as in that case we mimicked BID8 as closely as possible, as described above. All weight matrices are initialized using a normal distribution with a mean of 0 and a standard deviation of 1/ √ n h, where n h is the number of hidden units. All initial hidden states (for t < 1) are initialized to 0. For optimization, gradients are computed using full backpropagation through time, and we use stochastic gradient descent with a momentum of 0.9, with gradient clipping as described by BID31 at 1, and with a minibatch size of 100. Biases are generally initialized to 0, but we follow best practice for LSTM by initializing the forget-gate bias to 1 Gers et al. FORMULA1; BID21. For Clockwork RNNs, 8 exponential periods are used, as in the original paper. For MIST RNNs, 8 delays are used. We avoid manual learning-rate tuning in its entirety. Instead we run 50 trials for each experimental configuration. In each trial, the learning rate is drawn uniformly at random in log space between 10 −4 and 10 1, and initial weight matrices are also redrawn at random. We report over the top 10% of trials according to validationset error. (An alternative option is to report over all trials. However, because the majority of trials yields bad performance for all methods, this simply blurs comparisons. See for example FIG2 of BID16, which compares these two options.) Data preprocessing is kept minimal, with each input image individually shifted and scaled to have mean 0 and variance 1. We split the official training set into two parts, the first 58,000 used for training and the last 2,000 used for validation. Our test set is the same as the official test set, consisting of 10,000 images. Training is carried out by minimizing cross-entropy loss. In our experiments, the L relevant symbols are drawn at random (with replacement) from the set {0, 1, . . ., 9}; D is always a multiple of 10; and L is chosen to be D/10. This way the simplest baseline of always predicting the blank symbol yields a constant error rate for all experiments. No input preprocessing of any kind is performed. In each case, we generate 100,000 examples for training and 1,000 examples for validation. Training is carried out by minimizing cross-entropy loss. We use the same experimental setup as BID8, which currently holds state-of-the-art performance on these tasks. For kinematic inputs we use positions, velocities, and gripper angles for both hands. We also use their leave-one-user-out teset setup, with 8 users in the case of JIGSAWS and 15 users in the case of MISTIC-SL. Finally we use the same hyperparameters: 1 hidden layer of 1024 units; dropout with p = 0.5; 80 epochs of training with a learning rate of 1.0 for the first 40 epochs and having the learning rate every 5 epochs for the rest of training. As mentioned in the main paper, the primary difference is that we replaced their LSTM layer with our simple RNN, LSTM, or MIST RNN layer. Training is carried out by minimizing cross-entropy loss. We follow BID16 and extract 12 mel frequency cepstral coefficients plus energy every 10ms using 25ms Hamming windows and a pre-emphasis coefficient of 0.97. However we do not use derivatives, ing in 13 inputs per frame. Each input sequence is individually shifted and scaled to have mean 0 and variance 1 over each dimension. We form our splits according to BID17, ing in 3696 sequences for training, 400 sequences for validation, and 192 sequences for testing. Training is carried out by minimizing cross-entropy loss. Means and standard deviations are computed using the top 5 randomized trials out of 50 (ranked according to performance on the validation set). In BID4, emphasis was placed on hand-crafted features, and each subject was included during both training and testing (with no official test set defined). We instead operate on

represent the tldr for finding the corresponding scientific article

We introduce MIST RNNs, which a) exhibit superior vanishing-gradient properties in comparison to LSTM; b) improve performance substantially over LSTM and Clockwork RNNs on tasks requiring very long-term dependencies; and c) are much more efficient than previously-proposed NARX RNNs, with even fewer parameters and operations than LSTM.

Represent the TLDR for finding the corresponding scientific article

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.