Question: Represent the article for finding a single-sentence TLDR

Question: We present a method for policy learning to navigate indoor environments. We adopt a hierarchical policy approach, where two agents are trained to work in cohesion with one another to perform a complex navigation task. A Planner agent operates at a higher level and proposes sub-goals for an Executor agent. The Executor reports an embedding summary back to the Planner as additional side information at the end of its series of operations for the Planner's next sub-goal proposal. The end goal is generated by the environment and exposed to the Planner which then decides which set of sub-goals to propose to the Executor. We show that this Planner-Executor setup drastically increases the sample efficiency of our method over traditional single agent approaches, effectively mitigating the difficulty accompanying long series of actions with a sparse reward signal. On the challenging Habitat environment which requires navigating various realistic indoor environments, we demonstrate that our approach offers a significant improvement over prior work for navigation. The ability to model and understand the world at a high-level is crucial for performing complex tasks in real world environments. Part of this high-level understanding involves the ability to divide and plan out tasks that are complicated and have long time horizons into more manageable subtasks. For example, when navigating to a new location, we typically break the task down into a set of manageable directions (i.e. drive along a certain road until a familiar landmark before taking a turn). Imbuing machines with this ability of creating abstractions for long and complex tasks is an active area of research known as hierarchical learning (; 1999). Research for navigation has recently seen a rejuvenation due to the advent of learning-based approaches; ). Embodied learning-based approaches have shown some appealing properties over classical approaches such as being able to operate in complex environments with limited sensor data . However, there is a need for the ability to plan across long time horizons with sparse reward signals. This in effect, causes limitations such as the inability to overcome small obstacles when navigating towards a given goal and the requirement of invoking the environment a large number of times for any meaningful learning to occur . Works which have combined hierarchical reinforcement learning with imitation learning have shown promising (b;), by leveraging expert trajectories with policy sketches , which are less expensive to obtain; however these sketches still require annotation of the environment. In this work, we study such hierarchical control for the task of indoor navigation, whereby an embodied agent is randomly spawned within a novel and complex environment and must learn to navigate this environment through interaction (a). We address this challenging learning problem through a hierarchical policy approach, where two agents are cooperatively trained together. Each agent performs a different role, where one agent acts as a Planner, learning how to propose good sub-goals to an Executor agent, which acts at the low level to achieve these sub-goals (Fig. 1). In contrast to existing hierarchical policy learning approaches, communication between our two agents is two-way, where the Executor provides the Planner with a summary of its series of actions and recent observations. This aids the Planner in deciding the next sub-goal with additional side Figure 1: Our PLEX framework adopts a hierarchical policy approach, where a Planner proposes sub-goals for an Executor to act upon within an environment. The Planner receives an egocentric, top-down view with the target location and an embedding summary provided by the Executor. The Executor receives visual sensory data (i.e. colour and depth) as its input and a sub-goal provided by the Planner. Our method reduces the need for long-term planning and addresses the known sample inefficiency problem accompanying memory models within deep reinforcement learning approaches. information provided by the Executor. To this end, we propose PLEX, a planning and executing learning framework which offers the following contributions: • A hierarchical reinforcement learning approach where two agents specialise on different tasks but are jointly trained by sharing information • We demonstrate both theoretically and empirically that our method benefits from significantly improved sample efficiency as the time horizon is distributed between the Planner and Executor • By extension, our approach mitigates problems prevalent in long-horizon planning, especially those adopting LSTM planning approaches Hierarchical Reinforcement Learning The application of hierarchical reinforcement learning (; 1999;) in real world settings has allowed deep reinforcement learning approaches to scale with the increasingly higher sample requirements that are required by complex real world environments (b; a). Early works considered an options-based approach, where there was an underlying assumption of the existence of some useful set of options which are fully defined beforehand; this allowed learning to occur at a higher level in terms of those set of options (; 1999). , a hierarchical approach was adopted without the use of imitation learning. However, the work was limited to exploring non-realistic environments and information flow was uni-directional from master to controller. proposed sub-goals which have semantic meaning; although this demanded a high-level of supervision where rich annotations on a given environment were required. The work of showed that using a hierarchical learning approach can lead to a reduction in the cost of exploration on problems with sparse rewards under the limited context of game environments. Das et al. (2018b) explored the embodied question answering task (a), leveraging heuristics embedded in the environment for providing expert trajectories used in the imitation learning initialisation stage. This method is heavily supervised and assumes that full semantic information of the environment is known. Additionally, the approach of Das et al. (2018b) is limited to a specific set of sub-goals, for example: Exit-room, Find-object, etc., imposing limitations on the expressiveness of the master policy. By contrast, our method allows the Planner to propose sub-goals which directly relate to the given environment (i.e. a continuous point-goal vector) and does not rely on external annotations for supervision. Embodied Agent Learning There has been a recent surge of interest towards embodied agent learning where an agent is personified and spawned within an environment and set to complete a certain task. Such tasks may include question answering (a), point goal navigation , roaming and scene exploration or coverage . This problem is purposely set up in a way which allows for relatively easy transfer from simulation environment to a physical robotic platform operating in a real world environment. Various simulation environments have been designed around this (; ; ; ;), which aim to provide a realistic indoor simulation for training embodied agents on the aforementioned tasks. These environments allow an agent to invoke the environment numerous times, and in effect, scale the sample hungry nature of the problem to an extent where agents perform at a reasonable level (; a). trained an agent using the Proximal Policy Optimisation algorithm , augmenting it with a memory unit . Although this provided a strong initial baseline, achieving this required invoking an environment for millions of steps. The recent work of attempted to address this by proposing a memory component which makes use of the transformer network . Although this approach mitigated a key weakness in LSTM memory units , it incurs with it a linear growth of memory usage which is addressed via memory compression and hence increases the computational complexity of the model. The aforementioned issues are not present in our framework since the Executor's rollout is strictly shorter as we will soon demonstrate. 1 2. The sum of these i.i.d variables is S N = N i=1 X i. As we increase the rollout length N, the agent's probability of exploring decreases exponentially and is given by the following bound: Intuitively and as shown theoretically, reducing the rollout length N has a positive impact on the required exploration (for a more general form of this proposition, we refer the interested reader to). One way to exploit this insight is through a hierarchical approach which allows setting sub-goals that are closer in state space to the agent's current state. Given this insight, we now provide details of our hierarchical approach which consists of two main components: a Planner and an Executor policy. Each of these components is treated as an independent agent with respective policies π θ P L and π θ EX. A high-level overview of our framework is shown in Fig. 2. The environment is denoted as P, from which states s are sampled. A reward function R provides the reward conditioned on a state s, an action a and a goal. In general, a goal denoted by g indicates the end goal for the current episode, with g P denoting the sub-goals given by the Planner. N EX and N P L denote the respective rollout lengths of the Executor and Planner. Executor Building upon previous RL frameworks, an Executor is optimised using standard policy learning techniques . A key difference between existing non-hierarchical Figure 2: A system overview of PLEX which consists of two main components: a Planner and Executor agent. The Executor's task is to perform a series of actions such that it will traverse the environment towards a target location given by a point-goal vector (a sub-goal that is provided by the Planner). The Planner's task is to generate sub-goals that help navigate the entire system to an end goal (which only the Planner can see). An environment's state is comprised of an RGB-D observation, an egocentric map and a point-goal vector measurement pointing towards a target location as defined by. The Executor is provided with the RGB-D observation, a sub-goal generated by the Planner and returns an Executor Latent Information (ELI) vector that summarises its rollout. The Planner sees an egocentric map with a roughly 4m view horizon and the ELI vector. The Planner and Executor agents are trained simultaneously and using the sub-goal by the Planner and ELI vector by the Executor, we create a two-way communication channel, enabling our framework to regulate the flow of information between our two agents. approaches and our approach is the Executor End State or sub-goal g P, which is not provided by the environment but rather by a Planner policy. Consequently, the rewards provided to the Executor are sampled using the sub-goal (i.e.: r i ∼ R(r i |s i−1, a i, g P)). In line with this, the Executor policy also provides an Executor Latent Information (ELI) vector which summarises its rollout experience back to the Planner policy to provide feedback for the Planner towards planning the next sub-goal. In effect, this creates a cooperative feedback loop between the Executor and Planner. A detailed outline of the Executor routine is provided by Algorithm 1. Planner The Planner does not invoke the environment directly, but it modifies the goal and rewards observed by the Executor when the Executor interacts with the environment. This modification is achieved through the generation of planned sub-goals given to the Executor by the Planner. As such, given a state s and an ELI vector provided by the Executor, the Planner produces a subgoal. Using this sub-goal, the Executor routine (Algorithm 1) is invoked and a reward is returned to the Planner which provides it with feedback on the generated sub-goal. Upon optimising the Planner, we note that the Executor should not be directly affected from the Planner optimisation process (i.e. we terminate backpropagated gradients from the Planner policy before they reach the Executor policy). In other words, the ELI vector provided by the Executor does not affect the Executor directly, but implicitly through improving the Planner policy's generated sub-goals since providing direct feedback to the Executor through the ELI vector will in a redundant Planner. Algorithm 2 fully outlines our approach and details the Planner routine. We focus on the indoor PointGoal navigation task as defined in. In this setup, a point-goal vector providing the distance and angle towards the target point in the environment is assumed. If the indoor environment would be an empty space, this task would be trivial. However, in a more realistic, real world setting, the agent is spawned in a room within the environment and the target may be in a different room. In such a case, the agent needs to effectively navigate around obstacles and plan its future actions in the presence of other rooms that may or may not lead to the target point. The environment provides RGB-D sensory information and inaccurate access to the location and orientation (as could be obtained using a fusion of GPS and IMU sensors ). From these sensors, the environment can provide a coarse egocentric top-down map; we construct our egocentric map by adapting a similar approach in 1 (Section A.1 details the method of for constructing the egocentric map). The top-down map has a resolution of 0.5m per pixel which is a reasonable assumption given the provided sensors. In our hierarchical problem formulation, the Planner is provided with the egocentric, top-down map of size 32 × 32 pixels, the point-goal vector (translated to ego-map point-goal) (distance, direction) and the Executor Latent Information (ELI) provided by the Executor (a vector of 128 values). The Planner emits a binary category distribution for continue or stop operations and two continuous sub-goal distributions: the distance N (µ ρ, σ ρ) and direction N (µ θ, σ θ). The Executor is provided with RGB-D sensory information of 256 × 256 resolution along with the sub-goal computed by the Planner and emits a four category distribution with the actions described in Section 4.3, and emits the ELI vector which is returned to the Planner when the Executor stops or performs the maximum number of actions, N EX (note that for the Executor, the stop action does not stop the entire episode but simply returns control back to the Planner). Our Planner has a perception model for extracting embeddings from the egocentric map. This perception model is a CNN composed of 2 convolutional layers with filter size 4 × 4 and 3 × 3 with a stride of 2 and 1 respectively with ReLU activation. The output of these convolution layers is flattened and concatenated with the ELI vector and transferred to the last fullyconnected linear layer which outputs a 512 vector with ReLU activation. This output is concatenated with the 2-valued point-goal vector. The Executor's perception model consists of 3 convolution layers with filter sizes of 8 × 8, 4 × 4 and 3 × 3 with strides of 4, 2 and 1 respectively and ReLU activations. This is followed by a fullyconnected linear layer which outputs an embedding of 512 or 128, which depends on the model variant (LSTM or SMT respectively). We concatenate the 2-valued sub-goal vector to the output embedding. For the LSTM memory model, we employ the GRU formulation with hidden state of size 512. For the SMT memory model, we use an 8 multi-head attention mechanism (we do not employ the furthest point sampling technique used in as it did not affect the performance of our model). The output of either the GRU component or the SMT is the action embedding (a vector of 128 values corresponding to the output state vector of the LSTM) and also functions as the ELI vector. Both the Planner and the Executor policies are trained using the Proximal Policy Optimisation algorithm from , following the original settings found in the paper except for parameters specific to Algorithms 1 and 2: N EX = 10, γ EX = 0.9, N P L = 128, γ P L = 0.95. Note that the values of N EX and γ EX are chosen such that the Executor will focus on short-term rewards; this in turn will implicitly encourage the Planner to generate subsequent sub-goals which are nearby (further discussed in Section 4.4). Habitat is a modular environment API and can be used with realistic indoor environments coupled with tasks such as PointGoal navigation and Embodied Question Answering (a). For our experiments, we use the Gibson dataset and perform the PointGoal navigation task . We use the same train-test split as provided in which consists of 72 train scenes with 4.9M PointGoal navigation tasks and 16 unseen test scenes comprised of 1k PointGoal navigation tasks. The environment accepts 4 actions: [Forward, Turn left, Turn right, Stop], where invoking the stop action ends an episode with an episode running to a maximum of 500 steps otherwise. The reward structure is similar to with a success providing a reward of 10; otherwise, the agent's reward is the change in geodesic distance between the agent and the target with an additional slack of λ = −0.01. In addition, we penalise for obstacle collision with a cost c = −0.01. This additional collision penalty is a desirable property for embodied agents . In Section 3, we discussed the negative impact a long rollout horizon has on the exploration efficiency. The context was in a setup which allowed two actions, and did not take into account the complexities of realistic environments (such as obstacles and more actions). Hence, to choose an appropriate rollout length for the Executor (given the chosen environment and sensory input), we perform a short experiment which analyses a random exploration policy within the proposed setup. We randomly choose 500 starting locations from the training scenes, with a different initial starting point for recurring scenes. The agent can take 3 actions: [Forward, Turn Left, Turn Right], where the "Stop" action is excluded so that the rollout length can be inspected without being shortened. As we increase the number of random actions the random policy takes (N =), we inspect two values: the geodesic distance between the starting position and the end position of the policy (Fig. 3a), and the geodesic distance normalised with the number of random actions taken by the policy (Fig. 3b). The geodesic distance normalised by policy actions indicates the average distance travelled per number of actions and is an indication of exploration efficiency. As such, for initial exploration, we would generally want this value to be high. Fig. 3b observes that as the rollout horizon of the random policy increases, the exploration efficiency begins to rapidly decline. Whilst this decline is not exponential as suggested by Proposition 1, it provides us with further motivation not to set N EX too high. Further, we need to consider a large enough N EX to enable the Executor explore its environment. Given Figs. 3a and 3b and given a egocentric map with a resolution of 0.5m per pixel, N EX = 10 appears to maintain a good balance between the two criteria we wish to impose on the Executor: a high probability of exploring more than a unit distance in the egocentric map, along with high exploration efficiency. Baselines We compare our approach to two baseline methods PPO LSTM and PPO SMT. The former refers to the method found in , and the latter is a recent approach found in. For comparison, we employ two variants of our PLEX framework: PLEX LSTM and PLEX SMT. This comparison allows a fair comparison regarding the benefits toward improved sample efficiency that our framework offers for both compared baselines. Evaluation Metrics For evaluating the performance on the PointGoal task, we examine the reward each method obtains and the success ratio of each method. A success for an episode is defined when the agent provides a stop action and the geodesic shortest path distance between the agent and its end goal is less than 0.2m. We do not use the Shortest Path Length (SPL) as a evaluation metric, The observed success rate of each model. In this case, a success is defined as the agent issuing a stop action when the geodesic shortest path distance between the agent and its goal is less than 0.2m. Note that across both reward and success rate curves, the baseline methods exhibit a slow down in learning at around 15M environment steps whilst our method remains relatively robust and is able to maintain a steady learning rate. as defined by since maximising on this metric as an objective can in agents which exhibit undesirable behaviours (such as approaching too closely and colliding with obstacles). Instead, we observe the reward which is a unitless measure and penalises undesirable behaviours such as obstacle collision and moving in the wrong direction. This metric, along with success rates, provides good indicators for assessing performance on this task . Quantitative Results In Figs. 4a and 4b, we show the respective reward and success ratios of each method. Similarly, in Table 1, we outline the mean and variance of the final rewards and success ratios for all the methods after 30M environment steps. Each experiment was repeated 10 times across all methods. Note that the LSTM variant of our method (PLEX LSTM) achieves a success rate of 0.81 which is only slightly higher than the PPO LSTM reported in. The key difference is the significant lower number of environment steps (approximately half) that our method required for achieving a similar . The key idea behind the ELI vector is to provide additional information to the Planner that may not be observed in the egocentric map. Hence, the ELI vector is a mechanism that can be used by the Executor to relay a compressed representation of its previous observations to the Planner; assisting it towards generating sub-goals. Since the ELI vector is an integration of previous observations outputted by the Executor's LSTM component, we refer to it as a summary of the Executor's rollout. We adapt a hierarchical framework by , where conveniently, we can modify the Q-learning approach to a policy-based approach. In this context, our Executor replaces the Performance Metrics PPO SMT PPO LSTM Table 1: Avg. Reward and Avg. Success across the baseline models and our method applied to each method. Note that whilst the performance difference between the two baselines is small, both of our models outperform the baselines by a significant margin. Specifically, our PLEX LSTM variant shows a significant improvement over both baselines, achieving a higher than the baseline PPO LSTM model trained on more than double the number of environments steps. The performance of our method compared with PPO LSTM on close and far subsets of the test set. In this case, close goals were selected when mean geodesic shortest path distance was less than 5.76m and were classified as far goals otherwise. "Controller" and our Planner serves as the "Meta Controller". This allows us to assess the effect of not providing the Planner with an ELI vector. Empirically, these are shown in Fig. 5a. We can see that when the ELI vector is not provided, the communication channel from the Executor to the Planner is removed and we observe a decline in performance. This observation aligns well with our framework and demonstrates the value of enabling communication from the Executor back to the Planner. Across both reward and success ratio curves, the two baseline methods gradually exhibit a slow down behaviour starting at around 15M timesteps. In contrast, our PLEX framework appears relatively robust, demonstrating a more consistent learning pace. This highlights a key difference between our hierarchical method compared to the baseline approaches. Both baselines and our method are able to successfully learn simpler tasks, where the goal is relatively close to the agent's spawn location. However, for tasks where the goal is further away, the need for a more structured approach is necessary. Our empirical illustrate a potential drawback of LSTM-like units (and by extension, GRU), where they may suffer when learning long time horizon tasks. We divide the test set into two subsets containing close and far goals using a dividing criteria which uses the mean geodesic shortest path between the agent's spawn location and its end goal. In this case, close goals were selected when mean distance was less than 5.76m and were classified as far goals otherwise. The for this experiment are shown in Fig. 5b where the PPO LSTM is unable to navigate to the far goals in the allocated number of environment steps. In contrast, our method is far more successful in this task and is able to achieve a higher score on reaching far goals due to the extra reward provided for navigating to further away targets. Although the PPO LSTM baseline does eventually learn to navigate to further targets, it requires a much larger number of training iterations and environment invocations. In this work, we present a hierarchical reinforcement learning approach for solving PointGoal navigation tasks. Our proposed approach uses a cooperative learning strategy in which two agents, an Executor and a Planner are jointly learned to solve this task. This is enabled through a two-way communication channel established between the two agents through the use of an Executor Latent Information vector provided by the Executor and sub-goals generated by the Planner. We motivate the use of this hierarchical approach both theoretically, as well as through empirical experiments which demonstrate a significant improvement in sampling efficiency of our approach, allowing our structured approach to perform significantly better on increasingly harder tasks when compared to baseline approaches. Given an embodied agent trajectory of length N with corresponding depth images {D n} N n=1 ∈ R H×W and relative camera poses {T n} N n=1 ∈ SE, the set of 3D points P n ∈ R HW ×3 for frame n are extracted as follow: where the camera intrinsic matrix is K ∈ R 3×3. We then define the global set of all points at the end of the trajectory as: M ∈ R HW N ×3. The set M starts empty and is iteratively updated by the following rule: M n = M n−1 ∪ P n As the set of all points M is built, the egocentric map can be derived by projecting all points in M to a plane (we lose the "height" dimension and points projected to the same slot are resolved by taking the highest point), cropping and aligning a box using the vector of the agent's facing direction which in an egocentric map. Proof. The moment generating function of S N is given by: The probability of the Bernoulli sum being greater then αN with α > 1 2 can be bounded using the Chernoff bound: Analytically solving this bound, we obtain θ * = log α 1−α. We are specifically interested in the case where α > 1 2, and as a θ * > 0. Substituting θ * into Eq. 5 and defining β ≡ log 1 1−α + α log α 1−α concludes the proof. Algorithm 1 A pseudo algorithm of Executor Policy Gradient; Actor-Critic variant N EX ← initialise Executor rollout length γ EX ← initialise Executor discount factor g P ← initialise Planner goal r P = 0 initialise Planner reward buf f ers[s b, a b, r b] ← initialise state, action, reward buffers for i = 1 to N EX do a i, e θ EX ∼ π θ EX (a i, e θ EX |s i−1, g P) Sample policy for action and latent information s i, t i ∼ P(s i, t i |s i−1, a i) Sample environment for state and terminal r i ∼ R(r i |s i−1, a i, g P) Sample reward given Planner goal accumulate r P ← r ∼ R(r|s i−1, a i, g) Sample reward given end goal update buf f ers ← s i−1, a i, r i end for π θ EX ← update(π θ EX, buf f ers, γ EX) (as in PPO ) return (s i, r P, t i, e θ EX)

Represent the short summary for finding the corresponding scientific article (long):

We present a hierarchical learning framework for navigation within an embodied learning setting

The overall goal of this work is to enable sample-efficient imitation from expert demonstrations, both with and without the provision of expert action labels, through the use of f-divergences.


Question: Saliency methods aim to explain the predictions of deep neural networks. These methods lack reliability when the explanation is sensitive to factors that do not contribute to the model prediction. We use a simple and common pre-processing step ---adding a mean shift to the input data--- to show that a transformation with no effect on the model can cause numerous methods to incorrectly attribute. We define input invariance as the requirement that a saliency method mirror the sensitivity of the model with respect to transformations of the input. We show, through several examples, that saliency methods that do not satisfy a input invariance property are unreliable and can lead to misleading and inaccurate attribution. While considerable research has focused on discerning the decision process of neural networks BID1 BID9 BID2 BID15 BID12 BID0 BID14 BID6 BID5 BID16 BID13 BID11 BID4, there remains a trade-off between model complexity and interpretability. Research to address this tension is urgently needed; reliable explanations build trust with users, helps identify points of model failure, and removes barriers to entry for the deployment of deep neural networks in domains high stakes like health care and others. In deep models, data representation is delegated to the model. We cannot generally say in an informative way what led to a model prediction. Instead, saliency methods aim to infer insights about the f (x) learnt by the model by ranking the explanatory power of constituent inputs. While unified in purpose, these methods are surprisingly divergent and non-overlapping in outcome. Evaluating the reliability of these methods is complicated by a lack of ground truth, as ground truth would depend upon full transparency into how a model arrives at a decision -the very problem we are trying to solve for in the first place BID13 BID4.Given the need for a quantitative method of comparison, several properties such as completeness BID0 BID13, implementation invariance and sensitivity BID13 have been articulated as desirable to ensure that saliency methods are reliable. Implementation invariance, proposed as an axiom for attribution methods by BID13, is the requirement that functionally equivalent networks (models with different architectures but equal outputs all inputs), always attribute in an identical way. This work posits that a second invariance axiom, which we term input invariance, needs to be satisfied to ensure reliable interpretation of input contribution to the model prediction. Input invariance requires that the saliency method mirror the sensitivity of the model with respect to transformations of the input. We demonstrate that numerous methods do not satisfy input invariance using a simple transformation -mean shifts of the input-that does not affect model prediction or weights. We limit our treatment of input invariance to showing that there exist cases where this property is not satisfied and welcome future research on a broader treatment of this topic. In this work we:• introduce the axiom input invariance and demonstrate that certain saliency methods do not satisfy this property when considering a simple mean shift in the input. (See FIG3).• show that when input invariance is missing, the saliency method becomes unreliable and misleading. Using two example reference points for each method we demonstrate that changing the reference causes the attribution to diverge. The attributions are visualized by multiplying them with the input image as is done in the IG paper 1 BID13. Visualisations were made on ImageNet BID7 and the VGG16 architecture BID9.• demonstrate that "reference point" methods-Integrated gradients and the Deep Taylor Decomposition-have diverging attribution and input invariance breaking points that depends upon the choice of reference FIG0.In Section 2, we detail our experiment framework. In Section 3, we determine that while the model is invariant to the input transformation considered, several saliency methods attribute to the mean shift. In Section 4 we discuss "reference point" methods and illustrate the importance of choosing an appropriate reference before discussing some directions of future research in Section 5. We show that, by construction, the bias of a neural network compensates for the mean shift ing in two networks with identical weights and predictions. We first demonstrate this point and then describe the details of our experiment setup to evaluate the input invariance of a set of saliency methods. We compare the attribution across two networks, f 1 (x) and f 2 (x). f 1 (x) is a network trained on input x i 1 that denotes sample i from training set X 1. The classification task of network 1 is: DISPLAYFORM0 is a network that predicts the classification of a transformed input x ∀i, DISPLAYFORM1 Network 1 and 2 differ only by construction. Consider the first layer neuron before non-linearity in DISPLAYFORM2 We alter the biases in the first layer neuron by adding the mean shift m 2. This now becomes Network 2: DISPLAYFORM3 As a the first layer activations are the same for f 1 (x) and f 2 (x): DISPLAYFORM4 Note that the gradient with respect to the input remains unchanged as well: DISPLAYFORM5 We have shown that Network 2 cancels out the mean shift transformation. This means that f 1 (x) and f 2 (x) have identical weights and produce the same output for the corresponding samples, DISPLAYFORM6 In the implementation of this experimental framework, Network 1 is a 3 layer multi-layer perceptron with 1024 ReLu-activated neurons each. Network 1 classifies MNIST image inputs in a encoding. Network 2 classifies MNIST image inputs in a [-1,0] MNIST encoding. The first network is trained for 10 epochs using mini-batch stochastic gradient descent (SGD). The second network is created using the approach above. The final accuracy is 98.3% for both 3. In 3.1 we introduce key approaches to the classification of inputs as salient and the saliency methods we evaluate. In 3.2 we find that gradient and signal methods are input invariant. In 3.3 we find that most attribution methods considered have points where they start to break down. Most saliency research to date has centered on convolutional neural networks. These saliency methods broadly fall into three different categories:1. Gradients (Sensitivity) BID1 BID10 ) shows how a small change to the input affects the classification score for the output of interest.2. Signal methods such as DeConvNet BID15, Guided BackProp BID12 and PatternNet BID4 aim to isolate input patterns that stimulate neuron activation in higher layers.3. Attribution methods such as Deep-Taylor Decomposition BID5 and Integrated Gradients BID13 ) assign importance to input dimensions by decomposing the value y j at an output neuron j into contributions from the individual input dimensions: DISPLAYFORM0 s j is the decomposition into input contributions and has the same number of dimensions as x, A(x) j signifies the attribution method applied to output j for sample x. Attribution methods are distinguished from gradients by the insistence on completeness: the sum of all attributions should be approximately equal to the original output y i. We consider the input invariance of each category separately (by evaluating raw gradients, GuidedBackprop, Integrated Gradients and Deep Taylor Decomposition) and also benchmark the input invariance of SmoothGrad (BID11), a method that wraps around an underlying saliency approach and uses the addition of noise to produce a sharper visualization of the saliency heatmap. The experiment setup and methodology is as described in Section 2. Each method is evaluated by comparing the saliency heatmaps for the predictions of network 1 and 2, where x i 2 is simply the mean shifted input (x i 1 + m 2). A saliency method that is not input invariant will not produce identical saliency heatmap for Network 1 and 2 despite the mean shift of the input. Sensitivity and signal methods are not sensitive to the mean shift in inputs. In FIG2 raw gradients, PatternNet (PN, BID4) and Guided Backprop produce identical saliency heatmaps for both networks. Intuitively, gradient, PN and GB are input invariant given that we are comparing two networks with an identical f (x). Both methods determine attribution entirely as a function of the network/pattern weights and thus will be input invariant as long as we are comparing networks with identical weights. In the same manner, we can say that these methods will not be input invariant when comparing networks with different weights (even if we consider models with different architectures but identical predictions for every input). We evaluate the following attribution methods: gradient times input (GI), integrated gradients (IG, BID13) and the deep-taylor decomposition (DTD, BID5).In 3.3.1 we find GI to be sensitive to meaningless input shifts. In 3.3.2 we group discussion of IG and DTD under "reference point" methods because both require that attribution is done in reference to a defined point. We find that the choice of reference point can cause input invariance to become arbitrary. We find that the multiplication of raw gradients by the image breaks attribution reliability. In FIG3 GI produces different saliency heatmaps for both networks.. Gradient x Input, IG and DTD with a zero reference point, which is equivalent to LRP BID0 BID5, are not reliable and produce different attribution for each network. IG with a black image reference point and DTD with a PA reference point are not sensitive to the transformation of the input. In 3.2 we determined that a heatmap of gradients alone is not sensitive to the input transformation. GI multiplies the gradient w.r.t. the input with the input image. DISPLAYFORM0 Multiplying by the input means attribution is no longer reliable because the input shift is carried through to final attribution. Naive multiplication by the input, as noted by BID11, also constrains attribution without justification to inputs that are not 0. Both Integrated Gradients (IG, BID13) and Deep Taylor Decomposition (DTD, BID5) determine the importance of inputs relative to a reference point. DTD refers to this as the root point and IG terms the reference point a baseline. The choice of reference point is not determined a priori by the method and instead left to end user. The choice of reference point determines all subsequent attribution. In FIG0 IG and DTD show different attribution depending on the choice of reference point. We show that the certain reference point also cause IG and DTD to are not input invariant. Integrated gradients (IG) Integrated Gradients (IG, BID13) attribute the predicted score to each input with respect to a baseline x 0. This is achieved by constructing a set of inputs interpolating between the baseline and the input. DISPLAYFORM0 Since this integral cannot be computed analytically, it is approximated by a finite sum ranging over α ∈. DISPLAYFORM1 We evaluate whether two possible IG reference points satisfy input invariance. Firstly, we consider an image populated uniformly with the minimum pixel from the dataset (x 0 = min(x)) (black image) and a zero vector image. We find that IG attribution under certain reference points is not input invariant. In FIG3, IG with black reference point produces identical attribution heatmaps whereas IG with a zero vector reference point is not input invariant. IG using a black reference point is not sensitive to the mean input shift because x 0 = min(x) is determined after the mean shift of the input so the difference between x and x 0 remains the same for both networks. In network 1 this is (x 1) − min(x 1) and in network 2 this is (x 2 + m 2) − min(x 2 + m 2).IG with a zero vector reference point is not input invariant because while the difference in network 1 is (x 1 − x 0), the difference in network 2 becomes (x 2 + m 2) − x 0.Deep Taylor Decomposition (DTD) determines attribution relative to a reference point neuron. DTD can satisfy input invariant if the right reference point is chosen. In the general formulation, the attribution of an input neuron j is initialized to be equal to the output of that neuron. The attribution of other output neurons is set to zero. This attribution is backpropagated to its input neurons using the following distribution rule where s l j is the attribution assigned to neuron j in layer l: DISPLAYFORM2 We evaluate the input invariance of DTD using a reference point determined by Layer-wise Relevance Propagation (LRP) and PatternAttribution (PA). In FIG3, DTD satisfies input invariance when using a reference point defined by PA however it loses reliability when using a reference point defined by LRP.Layer-wise Relevance Propagation (LRP, BID0) is sensitive to the input shift because it is a case of DTD where a zero vector is chosen as the root point. 2. The back-propagation rule becomes: DISPLAYFORM3 s l−1,j depends only upon the input and so attribution will change between network 1 and 2 because x 1 andx 2 differ by a constant vector. PatternAttribution (PA) satisfies input invariance because the reference point x 0 is defined as the natural direction of variation in the data BID4. The natural direction of the data is determined based upon covariances and thus compensates explicitly for the mean in the data. Therefore it is by construction input invariant. The PA root point is: DISPLAYFORM4 where DISPLAYFORM5 In a linear model: DISPLAYFORM6 For neurons followed by a ReLu non-linearity the vector a accounts for the non-linearity and is computed as: DISPLAYFORM7.Here E + denotes the expectation taken over values where y is positive. Figure 4: Smoothgrad inherits the invariance properties of the underlying attribution method. SG is not sensitive to the input transformation for gradient and signal methods (SG-PA and and SG-GB). SG lacks input invariance for integrated gradients and deep taylor decomposition when a zero vector refernce point is used, but is not sensitive when PatternAttribution (SG-PA) or a black image (SG-Black) are used. SG is not input invariant for gradient x input. PA reduces to the following step: DISPLAYFORM8 The vector a depends upon covariances and thus removes the mean shift of the input. The attribution for both networks is identical. SmoothGrad replaces the input with N identical versions of the input with added random noise. These noisy inputs are injected into the underlying attribution method and final attribution is the average attribution across N. For example, if the underlying methods are gradients w.r.t. the input. g(x) j = ∂f (x)j ∂x SG becomes: DISPLAYFORM0 SG often in aesthetically sharper visualizations when applied to multi-layer neural networks with non-linearities. SG does not alter the attribution method itself so will always inherit the input Figure 5: Evaluation of attribution method sensitivity using MNIST. Gradient x Input, IG with both a black and zero reference point and DTD with a LRP reference point, do not satisfy input invariance and produce different attribution for each network. DTD with a PA reference point are not sensitive to the transformation of the input.invariance of the underlying method. In Fig. 4 applying SG on top of gradients and signal methods (PA and GB) produces identical saliency maps. SG is not input invariant when applied to gradient x input, LRP and zero vector reference points which compares SG heatmaps generated for all methods discussed so far. SG is not sensitive to the input transformation when applied to PA and a black image. IG and DTD do not satisfy input invariance under certain reference points. The reference point determines subsequent attribution. In FIG0 attribution visually diverges for the same method if multiple reference points are considered. A reasonable reference point for IG and DTD will naturally depend upon domain and task. Unintentional misrepresentation of the model is very possible when the choice of vantage point can lead to very different . Thus far, we have discussed attribution for image recognition tasks with the assumption that preprocessing steps are known and visual inspection of the points determined to be salient is possible. For Audio and Language based models where input interaction is more intricate, attribution becomes even more challenging. If we cannot determine the implications of reference point choice, we are limited in our ability to say anything about the reliability of the method. To demonstrate this point, we construct a constant shift of the input that takes advantage of attribution points of failure discussed thus far. Almost all methods are sensitive to this input transformation which in a misleading explanation of the model prediction. Network 1 is the same as introduced in Section 2. We consider a transformation x ∀i, DISPLAYFORM0 Network 2 is identical to network 1 by construction (see Section 2). Note that x In Fig. 5 all attribution methods except for PA are sensitive to this constant shift. The is that we are able to manipulate the attribution heatmap of an MNIST prediction so that the chosen samplê x appears. Using a black image as a reference point for IG no longer satisfies input invariance (as it did in the experiments in Section 3).The samplex can be any abitrary vector. We conduct the same experiment with a hand drawn kitten image. We construct m 2 by choosing a desired attributionŝ that should be assigned to a specific samplex when the gradient is multiplied with the input. We compute a m 2 that will ensure the specific x i 2 receives the desired attribution as follows: DISPLAYFORM1 To make sure that the original image is still recognizable as belonging to its class, we clip the shift to be within [-.3,.3]. Of course, the attributions of the other samples in the dataset is impacted too. In FIG6 we see that we are again able to purposefully misrepresent the explanation of the model prediction. It is important to note that that some of these methods would have satisfied input invariance if the data had been normalized prior to attribution. For example, IG with a black baseline will satisfy input invariance if the data is always normalized. However, this is far from a systematic treatment of the reference point selection and there are cases outside of our experiment scope where this would not be sufficient. We believe an open research question is furthering the understanding of reference point choice that guarantee reliability without relying on case-by-case solutions. Saliency methods are powerful tools to gain intuition about our model. We consider some examples that can cause a break in the reliability of these methods. We show that we are able to purposefully create a deceptive explanation of the network using a hand drawn kitten image. We introduce input invariance as a prerequisite for reliable attribution. Our treatment of input invariance is restricted to demonstrating there is at least one input transformation that causes attribution to fail. We hope this work drives further discussion on this subject. We also acknowledge that saliency methods may still provide intuition for image recognition tasks even if they are not input invariant. Our work is motivated in part because while we can visually inspect for catasthropic attribution failure in images, other modalities (like audio or word vectors) are more opaque and prone to unintentional misrepresentation.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR

Question: Large Transformer models routinely achieve state-of-the-art on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O(L^2) to O(L), where L is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of N times, where N is the number of layers. The ing model, the Reformer, performs on par with Transformer models while being much more memory-efficient and much faster on long sequences. The Transformer architecture is widely used in natural language processing and yields state-of-the-art on a number of tasks. To obtain these , researchers have resorted to training ever larger Transformer models. The number of parameters exceeds 0.5B per layer in the largest configuration reported in while the number of layers goes up to 64 in . Transformer models are also used on increasingly long sequences. Up to 11 thousand tokens of text in a single example were processed in and when processing other modalities, like music and images, even longer sequences are commonplace. These large-scale long-sequence models yield great but strain resources to the point where some argue that this trend is breaking NLP research 1. Many large Transformer models can only realistically be trained in large industrial research laboratories and such models trained with model parallelism cannot even be fine-tuned on a single GPU as their memory requirements demand a multi-accelerator hardware setup even for a single training step. Do large Transformer models fundamentally require such huge resources or are they simply inefficient? Consider the following calculation: the 0.5B parameters used in the largest reported Transformer layer account for 2GB of memory. Activations for 64K tokens with embedding size 1024 and batch size 8 account for 64K × 1K × 8 = 0.5B floats, requiring another 2GB of memory. If our memory use was only per-layer, then we should fairly easily fit a large Transformer even on sequences of length 64K on a single accelerator. Further, the whole corpus used to train BERT only requires 17GB to store. Why is it then that we cannot even fine-tune these models on single machines? The above estimate includes only per-layer memory and input activations cost and does not take into account the following major sources of memory use in the Transformer. • Memory in a model with N layers is N -times larger than in a single-layer model due to the fact that activations need to be stored for back-propagation. • Since the depth d f f of intermediate feed-forward layers is often much larger than the depth d model of attention activations, it accounts for a large fraction of memory use. • Attention on sequences of length L is O(L 2) in both computational and memory complexity, so even for a single sequence of 64K tokens can exhaust accelerator memory. We introduce the Reformer model which solves these problems using the following techniques: 1 https://hackingsemantics.xyz/2019/leaderboards/ • Reversible layers, first introduced in , enable storing only a single copy of activations in the whole model, so the N factor disappears. • Splitting activations inside feed-forward layers and processing them in chunks removes the d f f factor and saves memory inside feed-forward layers. • Approximate attention computation based on locality-sensitive hashing replaces the O(L 2) factor in attention layers with O(L) and so allows operating on long sequences. We study these techniques and show that they have negligible impact on the training process compared to the standard Transformer. Splitting activations in fact only affects the implementation; it is numerically identical to the layers used in the Transformer. Applying reversible residuals instead of the standard ones does change the model but has a negligible effect on training in all configurations we experimented with. Finally, locality-sensitive hashing in attention is a more major change that can influence the training dynamics, depending on the number of concurrent hashes used. We study this parameter and find a value which is both efficient to use and yields very close to full attention. We experiment on a synthetic task, a text task (enwik8) with sequences of length 64K and an image generation task (imagenet-64 generation) with sequences of length 12K. In both cases we show that Reformer matches the obtained with full Transformer but runs much faster, especially on the text task, and with orders of magnitude better memory efficiency. Dot-product attention. The standard attention used in the Transformer is the scaled dot-product attention . The input consists of queries and keys of dimension d k, and values of dimension d v. The dot products of the query with all keys are computed, scaled by √ d k, and a softmax function is applied to obtain the weights on the values. In practice, the attention function on a set of queries is computed simultaneously, packed together into a matrix Q. Assuming the keys and values are also packed together into matrices K and V, the matrix of outputs is defined as: Multi-head attention. length, length]. In the experimental section we train a model on sequences of length 64K -in this case, even at batch-size of 1, this is a 64K × 64K matrix, which in 32-bit floats would take 16GB of memory. This is impractical and has hindered the use of the Transformer for long sequences. But it is important to note that the QK T matrix does not need to be fully materialized in memory. The attention can indeed be computed for each query q i separately, only calculating softmax(V once in memory, and then re-computing it on the backward pass when needed for gradients. This way of computing attention may be less efficient but it only uses memory proportional to length. We use this memory-efficient implementation of attention to run the full-attention baselines presented in the experimental section. Where do Q, K, V come from? The multi-head attention described above operates on keys, queries and values, but usually we are only given a single tensor of activations A of the shape [batch size, length, d model] -e.g., coming from embedding the tokens in a sentence into vectors. To build Q, K and V from A, the Transformer uses 3 different linear layers projecting A into Q, K and V with different parameters. For models with LSH attention, we want queries and keys (Q and Figure 1 : An angular locality sensitive hash uses random rotations of spherically projected points to establish buckets by an argmax over signed axes projections. In this highly simplified 2D depiction, two points x and y are unlikely to share the same hash buckets (above) for the three different angular hashes unless their spherical projections are close to one another (below). K) to be identical. This is easily achieved by using the same linear layer to go from A to Q and K, and a separate one for V. We call a model that behaves like this a shared-QK Transformer. It turns out that sharing QK does not affect the performance of Transformer, even if we additionally normalize the length of the keys K, as we show in the experimental Section 5. Hashing attention. For the LSH attention, we start with two tensors, Q=K and V of the shape [batch size, length, d model]. We keep the multi-head mechanism intact and focus on the attention computation from Equation 1. As already mentioned, the main issue is the term QK T, which has the shape [batch size, length, length]. But note that we are actually only interested in softmax(QK T). Since softmax is dominated by the largest elements, for each query q i we only need to focus on the keys in K that are closest to q i. For example, if K is of length 64K, for each q i we could only consider a small subset of, say, the 32 or 64 closest keys. That is much more efficient, but how can we find the nearest neighbors among the keys? Locality sensitive hashing. The problem of finding nearest neighbors quickly in high-dimensional spaces can be solved by locality-sensitive hashing (LSH). A hashing scheme that assigns each vector x to a hash h(x) is called locality-sensitive if nearby vectors get the same hash with high probability and distant ones do not. In our case, we actually only require that nearby vectors get the same hash with high probability and that hash-buckets are of similar size with high probability. We achieve this by employing random projections as follows (see Figure 1). To get b hashes, we first fix a random matrix R of size [d k, b/2]. We then define h(x) = arg max([xR; −xR]) where [u; v] denotes the concatenation of two vectors. This method is a known LSH scheme and is easy to implement and apply to batches of vectors. LSH attention. Knowing our LSH scheme and the general idea of hashing attention, we will now formalize the LSH attention we use in this paper. We first rewrite the equation for normal attention,, for a single query position i at a time: We introduce the notation P i to represent the set that the query at position i attends to, and z to denote the partition function (i.e. the normalizing term in the softmax). For clarity, we also omit scaling by For batching purposes we typically perform attention over a larger set P i = {0, 1, . . ., l} ⊇ P i while masking out elements not in P i: Now we turn to LSH attention, which we can think of in terms of restricting the set P i of target items a query position i can attend to, by only allowing attention within a single hash bucket. Figure 2 (a-b) shows a schematic comparison of full-attention with a hashed variant. Part (a) depicts that the attention matrix for full attention is typically sparse, but the computation does not take advantage of this sparsity. In (b), the queries and keys have been sorted according to their hash bucket. Since similar items fall in the same bucket with high probability, the full attention pattern can be approximated by only allowing attention within each bucket. Hash buckets in this formulation tend to be uneven in size, which makes it difficult to batch across buckets. Moreover, the number of queries and the number of keys within a bucket may be unequalin fact, it is possible for a bucket to contain many queries but no keys. To alleviate these issues, we first ensure that h(k j) = h(q j) by setting k j = qj qj. Next, we sort the queries by bucket number and, within each bucket, by sequence position; this defines a permutation where i → s i after sorting. In the sorted attention matrix, pairs from the same bucket will cluster near the diagonal (as depicted in Figure 2c). We can follow a batching approach where chunks of m consecutive queries (after sorting) attend to each other, and one chunk back (Figure 2d). Following our earlier notation, this corresponds to setting: If max i |P i | < m, then P i ⊆ P i. In practice we set m = 2l n buckets (where l is the sequence length). The average bucket size is l n buckets, and we assume that the probability of a bucket growing to twice that size is sufficiently low. The overall process of LSH attention is summarized in Figure 2. Multi-round LSH attention. With hashing, there is always a small probability that similar items nevertheless fall in different buckets. This probability can be reduced by doing multiple rounds of hashing with n rounds distinct hash functions {h, h,...}, such that: The multi-round case essentially involves performing LSH attention n rounds times in parallel; the details of the procedure are described in in Appendix A. Causal masking for shared-QK attention. In a Transformer decoder, masking (denoted by m(j, P i) in Equation 3 ) is used to prevent positions from attending into the future. To implement masking in LSH attention, we associate every query/key vector with a position index, re-order the position indices using the same permutations used to sort the query/key vectors, and then use a comparison operation to compute the mask. While attention to the future is not allowed, typical implementations of the Transformer do allow a position to attend to itself. Such behavior is undesirable in a shared-QK formulation because the dot-product of a query vector with itself will almost always be greater than the dot product of a query vector with a vector at another position. We therefore modify the masking to forbid a token from attending to itself, except in situations where a token has no other valid attention targets (e.g. the first token in a sequence). To verify the performance of LSH attention and study its behavior, we start with the following synthetic task: duplicate a sequence of symbols. In this task, each training and testing example has the form 0w0w where w ∈ {1, . . ., N} * is a sequence of symbols ranging from 1 to N (we use N = 127 in our experiments). An example with the word w of length 3 is given below. To study LSH attention, we train a language model on examples of the above form where each w is of length 511 (so the whole input 0w0w is of length 1024). As this is a language modeling task, we always predict the next symbol given all the previous ones, but we mask the loss and accuracy to only consider positions in the second half of the input, i.e., those that can actually be predicted. The above task can be solved perfectly (to accuracy 100% and loss 0) by a 1-layer Transformer model. Note though, that it requires non-local attention lookups, so it cannot be solved by any model relying on sparse attention with a limited span. To make it easy and fast to train but similar to models used in NLP, we use a 1-layer Transformer with d model = d f f = 256 and 4 heads. We train it for 150K steps in 4 different settings: with full attention, LSH attention with n rounds = 1, n rounds = 2 and n rounds = 4. From the summarized in Table 2 we see that a model trained with full attention can be immediately used with LSH attention, but at some loss of accuracy. When trained from scratch with LSH attention, the model trained with 4 hashes achieves almost perfect accuracy as well. Interestingly, the accuracy becomes perfect when evaluated with 8 hashes. It goes down when evaluated with 2 or 1 hashes. Models trained with less hashes show worse but even the model trained with just 1 hash performs almost perfectly when evaluated with 8 hashes. As the above section shows, the complexity of attention can be reduced from square in length to linear, provided an approximation is acceptable. But it is clear from Table 1 that each field starts with a b · n h · l term: the b · n h · l · d k, or alternatively b · l · d model cost cannot be avoided. Indeed, the activations before each layer are already of the size b · l · d model, so the memory use of the whole model with n l layers is at least b · l · d model · n l. Even worse: inside the feed-forward layers of Transformer this goes up to b · l · d f f · n l. In a big Transformer it is usual to set d f f = 4K and n l = 16 so with l = 64K this again would use an impractical 16GB of memory In this section, we show how to reduce this cost by first dealing with the n l part of the term using reversible layers and then showing how chunking can allow us to handle the d f f problem. RevNets. Reversible residual networks were introduced by where it was shown that they can replace ResNets for image classification. The main idea is to allow the activations at any given layer to be recovered from the activations at the following layer, using only the model parameters. Rather than having to checkpoint intermediate values for use in the backward pass, layers can be reversed one-by-one as back-propagation proceeds from the output of the network to its input. Whereas a normal residual layer performs a function x → y that operates on a single input and produces a single output and has the form y = x + F (x), a reversible layer works on pairs of inputs/outputs: (x 1, x 2) → (y 1, y 2), and follows the equations: A layer can be reversed by subtracting (rather than adding) the residuals: Reversible Transformer. We apply the RevNet idea to the Transformer by combining the attention and feed-forward layers inside the revnet block. In the notation above, F becomes an attention layer while G becomes the feed-forward layer. Note that Layer Normalization is moved inside the residual blocks. The reversible Transformer does not need to store activations in each layer and so gets rid of the n l term. In Section 5 we show that it performs the same as the normal Transformer when using the same number of parameters; we achieve this by having both x 1 and x 2 have size d model. Chunking. While reversibility covers the n l term, the thicker layers can still use a lot of memory. The feed-forward layer in particular can use intermediate vectors of dimensionality d f f = 4K or higher. However, computations in feed-forward layers are completely independent across positions in a sequence, so the computation can be split into c chunks: This layer is typically batched by performing operations for all positions in parallel, but operating on one chunk at a time can reduce memory. The reverse computation in and the backward pass are also chunked. In addition to the feed-forward layers, for models with large vocabulary (more than d model word types) we also chunk the log-probabilities at the output and calculate the loss for sections of the sequence at a time. Chunking, large batches and parameter reuse. With chunking and reversible layers the memory we use for activations in the whole network is independent of the number of layers. The same is not true for parameters though as their number grows with the number of layers. This problem is remedied though because we can swap layer parameters to and from CPU memory when this layer is not computing. In a standard Transformer this would be inefficient because memory transfer to CPU is slow. The batch size multiplied by length in Reformer is much larger though and therefore the amount of compute done with the parameters amortizes the cost of their transfer. The Transformer model introduced in has been used widely in natural language tasks and further extended to model diverse data such as music scores , and images ). Most notably, this model class has We assume n c = l/32 so 4l/n c = 128 and we write c = 128 2. max(bld f f, bn h ln r c)n l (bld f f + bn h n r lc)n l Reformer max(bld model, bn h ln r c) (bld f f + bn h n r lc)n l been applied successfully in the self-supervised training of extremely large language models (; . Given the enormous computational requirements of state of the art sequence models, there has been increasing interest in finding methods to reduce the memory footprint and computational requirements of Transformer models. In addition to standard methods such as precision reduction and gradient checkpointing , more efficient versions of the Transformer model's self-attention mechanism (a; b) have also recently been explored. In particular, leveraging sparsity in the attention layers has proved fruitful. OpenAI introduced the sparse Transformer which exploits a factorized sparse representation of attention. Using product-key attention to increase the key space has also been used to reduce memory requirements in the feed-forward layers with no loss in performance . Locality-sensitive hashing (LSH) has, to our knowledge, not been directly applied to Transformer attention layers before. But previous work using external memory with neural networks has dealt with memories of large sizes. The original implementation of memory networks and later work on scaling it ) used memory with size in the millions. The cost of doing so is that the memory must be fixed prior to training. Moreover, since during the beginning of training the model is unlikely to query the memory correctly, strong supervision is used to encourage the model to query memory locations that are useful. These hints are either given as additional supervising information by the task or determined heuristically as in. The requirement that the memory be fixed before has been removed in at the cost of memory size and later alleviated by. The last paper considered memory lookups with approximate nearest neighbors including both LSH and random kd-trees, but only for lookups in external memory. In this section we present experimental demonstrating the techniques described above. We analyze the techniques one-by-one to make clear which combinations have impact on performance. We start by showing that reversible layers and shared query-key spaces do not impact performance, then proceed to analyze hashing attention and finally the full Reformer model. We ran our experiments on the imagenet64 and enwik8-64K tasks, where the latter is a variant of enwik8 that is chunked into subsequences of 2 16 = 64K tokens. We use 3-layer models for our ablations so as to make it tractable to compare with the regular Transformer, which has high memory usage and performs full O(l 2) attention. All experiments have d model = 1024, d f f = 4096, and n heads = 8. They were trained with batch sizes of one sequence per GPU, with a total of 8 GPUs operating in parallel. We used the Adafactor optimizer for training our models. Code for training our models will be made publicly available. Effect of sharing QK. We first consider the effect of shared-QK attention on a regular Transformer model. Shared-QK attention sets k j = qj qj and prevents tokens from attending to themselves (except when no other context is available). In the left part of Figure 3, we plot perplexity curves for both regular and shared-QK attention. A shared query-key space does not perform worse than Figure 3: Effect of shared query-key space (left) and reversibility (right) on performance on enwik8 and imagenet64 training. The curves show bits per dim on held-out data. regular attention; in fact, for enwik8 it appears to train slightly faster. In other words, we are not sacrificing accuracy by switching to shared-QK attention. Effect of reversible layers. In the two plots on the right in Figure 3, we compare a regular Transformer per with the reversible one describe in Section 3. The two models have identical parameter counts, and the learning curves likewise appear to be nearly the same. These show that the memory savings in the reversible Transformer do not come at the expense of accuracy. LSH attention in Transformer. LSH attention is an approximation for full attention that, as evidenced in Figure 4, becomes more accurate as the number of hashes increases. At n rounds = 8, it already almost matches full attention. The computational cost of a model grows with the number of hashes, so this hyperparameter can be adjusted depending on the available compute budget. Additionally, as in Table 2, the number of hashes can be increased at evaluation time to produce more accurate . On the right half of Figure 5, we plot the speed of different attention types vs. the sequence length, while holding the total number of tokens fixed. We see that while regular attention becomes slower at longer sequence lengeth, LSH attention speed remains flat. Large Reformer models. To verify that the Reformer can indeed fit large models on a single core and train fast on long sequences, we train up to 20-layer big Reformers on enwik8 and imagenet64. As can be seen in Figure 5, these models fit into memory and train. We were not able to train Transformer baselines in this case as they are too slow and memory-hungry, but we see clear improvement with the number of layers. A 12-layer model on enwik8 trained for 20K steps with a dropout rate of 0.1 achieves 1.19 bits/dim on the test set. We also trained a 12-layer Reformer model for longer with further tuning and improvements and we reached 1.05 bits/dim on the enwiki8 test set 3. Reformer combines the modeling capacity of a Transformer with an architecture that can be executed efficiently on long sequences and with small memory use even for models with a large number of layers. We believe that this will help large, richly-parameterized Transformer models become more widespread and accessible. Also, the ability to handle long sequences opens the way for the use of the Reformer on many generative tasks. In addition to generating very long coherent text, the Reformer can bring the power of Transformer models to other domains like time-series forecasting, music, image and video generation. In this section we describe in more detail the multi-hash version of our LSH attention mechanism. We first repeat Equation from the main text, which describes a general formulation of attention with sparsity: exp (q i · k i − m(j, P i) − z(i, P i)) v j where m(j, P i) = ∞ if j / ∈ P i 0 otherwise In the multi-round case, a query position i can attend to key positions P i as defined in, which we also repeat here: For batching purposes, attention is performed on chunks of sorted queries/keys: Combining and gives: i that can be computed independently from other rounds, except for the inclusion of a term N i,j to avoid double-counting elements when constructing the union of P (r) i sets. In our implementation we fold the N i,j factor into the masking term m We also modify m (r) i,j to introduce a special case for i = j. This case is added because causal masking in a standard Transformer allows position i to attend to itself, which is not desirable in a shared-QK formulation. We set the mask to a large but finite value to disallow attention-in-place, except in the situation where a token has no other valid attention targets. For example, the first token in a sequence attends only to itself, because no prior context is available.

Represent the short summary for finding the corresponding scientific article (long)

Efficient Transformer with locality-sensitive hashing and reversible layers

represent the tldr for finding the corresponding scientific article

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.