Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In visual planning (VP), an agent learns to plan goal-directed behavior from observations of a dynamical system obtained offline, e.g., images obtained from self-supervised robot interaction. VP algorithms essentially combine data-driven perception and planning, and are important for robotic manipulation and navigation domains, among others. A recent and promising approach to VP is the semi-parametric topological memory (SPTM) method, where image samples are treated as nodes in a graph, and the connectivity in the graph is learned using deep image classification. Thus, the learned graph represents the topological connectivity of the data, and planning can be performed using conventional graph search methods. However, training SPTM necessitates a suitable loss function for the connectivity classifier, which requires non-trivial manual tuning. More importantly, SPTM is constricted in its ability to generalize to changes in the domain, as its graph is constructed from direct observations and thus requires collecting new samples for planning. In this paper, we propose Hallucinative Topological Memory (HTM), which overcomes these shortcomings. In HTM, instead of training a discriminative classifier we train an energy function using contrastive predictive coding. In addition, we learn a conditional VAE model that generates samples given a context image of the domain, and use these hallucinated samples for building the connectivity graph, allowing for zero-shot generalization to domain changes. In simulated domains, HTM outperforms conventional SPTM and visual foresight methods in terms of both plan quality and success in long-horizon planning. For robots to operate in unstructured environments such as homes and hospitals, they need to manipulate objects and solve complex tasks as they perceive the physical world. While task planning and object manipulation have been studied in the classical AI paradigm, most successes have relied on a human-designed state representation and perception, which can be challenging to obtain in unstructured domains. While high-dimensional sensory input such as images can be easy to acquire, planning using raw percepts is challenging. This has motivated the investigation of datadriven approaches for robotic manipulation. For example, deep reinforcement learning (RL) has made impressive progress in handling high-dimensional sensory inputs and solving complex tasks in recent years. One of the main challenges in deploying deep RL methods in human-centric environment is interpretability. For example, before executing a potentially dangerous task, it would be desirable to visualize what the robot is planning to do step by step, and intervene if necessary. Addressing both data-driven modeling and interpretability, the visual planning (VP) paradigm seeks to learn a model of the environment from raw perception and then produce a visual plan of solving a task before actually executing a robot action. Recently, several studies in manipulation and navigation have investigated VP approaches that first learn what is possible to do in a particular environment by self-supervised interaction, and then use the learned model to generate a visual plan from the current state to the goal, and finally apply visual servoing to follow the plan. One particularly promising approach to VP is the semi-parametric topological memory (SPTM) method proposed by Savinov et al.. In SPTM, images collected offline are treated as nodes in a graph and represent the possible states of the system. To connect nodes in this graph, an image classifier is trained to predict whether pairs of images were'close' in the data or not, effectively learning which image transitions are feasible in a small number of steps. The SPTM graph can then be used to generate a visual plan -a sequence of images between a pair of start and goal images -by directly searching the graph. SPTM has several advantages, such as producing highly interpretable visual plans and the ability to plan long-horizon behavior. However, since SPTM builds the visual plan directly from images in the data, when the environment changes -for example, the lighting varies, the camera is slightly moved, or other objects are displaced -SPTM requires recollecting images in the new environment; in this sense, SPTM does not generalize in a zero-shot sense. Additionally, similar to, we find that training the graph connectivity classifier as originally proposed by requires extensive manual tuning. Figure 1: HTM illustration. Top left: data collection. In this illustration, the task is to move a green object between gray obstacles. Data consists of multiple obstacle configurations (contexts), and images of random movement of the object in each configuration. Bottom left: the elements of HTM. A CVAE is trained to hallucinate images of the object and obstacles conditioned on the obstacle image context. A connectivity energy model is trained to score pairs of images based on the feasibility of their transition. Right: HTM visual planning. Given a new context image and a pair of start and goal images, we first use the CVAE to hallucinate possible images of the object and obstacles. Then, a connectivity graph (blue dotted lines) is computed based on the connectivity energy, and we plan for the shortest path from start to goal on this graph (orange solid line). For executing the plan, a visual servoing controller is later used to track the image sequence. In this work, we propose to improve both the robustness and zero-shot generalization of SPTM. To tackle the issue of generalization, we assume that the environment is described using some context vector, which can be an image of the domain or any other observation data that contains enough information to extract a plan (see Figure 1 top left). We then train a conditional generative model that hallucinates possible states of the domain conditioned on the context vector. Thus, given an unseen context, the generative model hallucinates exploration data without requiring actual exploration. When building the connectivity graph with these hallucinated images, we replace the vanilla classifier used in SPTM with an energy-based model that employs a contrastive loss. We show that this alteration drastically improves planning robustness and quality. Finally, for planning, instead of connecting nodes in the graph according to an arbitrary threshold of the connectivity classifier, as in SPTM, we cast the planning as an inference problem, and efficiently search for the shortest path in a graph with weights proportional to the inverse of a proximity score from our energy model. Empirically, we demonstrate that this provides much smoother plans and barely requires any hyperparameter tuning. We term our approach Hallucinative Topological Memory (HTM). A visual overview of our algorithm is presented in Figure 1. We evaluate our method on a set of simulated VP problems of moving an object between obstacles, which require long-horizon planning. In contrast with prior work, which only focused on the success of the method in executing a task, here we also measure the interpretability of visual planning, through mean opinion scores of features such as image fidelity and feasibility of the image sequence. In both measures, HTM outperforms state-of-the-art data-driven approaches such as visual foresight and the original SPTM. Context-Conditional Visual Planning and Acting (VPA) Problem. We consider the contextconditional visual planning problem from. Consider deterministic and fully-observable environments E 1,..., E N that are sampled from an environment distribution P E. Each environment E i can be described by a context vector c i that entirely defines the dynamics o Figure 1, the context could represent an image of the obstacle positions, which is enough to predict the possible movement of objects in the domain. 1 As is typical in VP problems, we assume our data D = {o Ti, c i} i∈{1,...,N} is collected in a self-supervised manner, and that in each environment E i, the observation distribution is defined as P o (·|c i). At test time, we are presented with a new environment, its corresponding context vector c, and a pair of start and goal observations o start, o goal. Our goal is to use the training data to build a planner Q h (o start, o goal, c) and an h-horizon policy π h. The planner's task is to generate a sequence of observations between o start and o goal, in which any two consecutive observations are reachable within h time steps. The policy takes as input the image sequence and outputs a control policy that transitions the system from o start to o goal. As the problem requires a full plan given only a context image in the new environment, the planner must be capable of zero-shot generalization. Note that the planner and policy form an interpretable planning method that allows us to evaluate their performance separately. For simplicity we will omit the subscript h for the planner and the policy. Semi-Parametric Topological Memory (SPTM) is a visual planning method that can be used to solve a special case of VPA. where there is only a single training environment, E and no context image. SPTM builds a memory-based planner and an inverse-model controller. At training, a classifier R is trained to map two observation images o i, o j to a score ∈ representing the feasibility of the transition, where images that are ≤ h steps apart are labeled positive and images that are ≥ l are negative. The policy is trained as an inverse model L, mapping a pair of observation images o i, o j to an appropriate action a that transitions the system from o i to o j. Given an unseen environment E *, new observations are manually collected and organized as nodes in a graph G. Edges in the graph connect observations o i, o j if R(o i, o j) ≥ s shortcut, where s shortcut is a manually defined threshold. To plan, given start and goal observations o start and o goal, SPTM first uses R to localize, i.e., find the closest nodes in G to o start and o goal. A path is found by running Dijkstra's algorithm, and the method then selects a waypoint o wi on the path which represents the farthest observation that is still feasible under R. Since both the current localized state o i and its waypoint o wi are in the observation space, we can directly apply the inverse model and take the action a i where a i = L(o i, o wi). After localizing to the new observation state reached by a i, SPTM repeats the process until the node closest to o goal is reached. is a deep generative model that can be used for learning a high-dimensional conditional distribution P o (·|c). The CVAE is trained by maximizing the evidence lower bound (ELBO):, where q θ (z|o, c) is the encoder that maps observations and contexts to the latent distribution, p θ (o|z, c) is the decoder that maps latents and contexts to the observation distribution, and r ψ (z|c) is the prior that maps contexts to latent prior distributions. Together p θ, q φ, r ψ are trained to maximize the variational lower bound above. We assume that the prior and the encoder are Gaussian, which allows the D KL term to be computed in closed-form. Monte-Carlo sampling and the reparametrization trick are used to approximate the gradient of the loss. Contrastive Predictive Coding (CPC) extracts compact representations that maximize the causal and predictive aspects of high-dimensional sequential data. A non-linear encoder g enc encodes the observation o t to a latent representation z t = g enc (o t). We maximize the mutual information between the latent representation z t and future observation o t+k with a log-. This model is trained to be proportional to the density ratio p(o t+k |z t)/p(o t+k) by the CPC loss function: the cross entropy loss of correctly classifying a positive sample from a set X = {o 1, ..., o N} of N random samples with 1 positive sample from p(o t+k |z t) and N − 1 negative samples sampled from p(o t+k): SPTM has been shown to solve long-horizon planning problems such as navigation from first-person view. However, SPTM is not zero-shot: even a small change to the training environment requires collecting substantial exploration data for building the planning graph. This can be a limitation in practice, especially in robotic domains, as any interaction with the environment requires robot time, and exploring a new environment can be challenging (indeed, applied manual exploration). In addition, similarly to, we found that training the connectivity classifier as proposed in requires extensive hyperparameter tuning. In this section, we propose an extension of SPTM to overcome these two challenges by employing three ideas - using a CVAE to hallucinate samples in a zero-shot setting, using contrastive loss for a more robust score function and planner, and planning based on an approximate maximum likelihood formulation of the shortest path under uniform state distribution. We call this approach Hallucinative Topological Memory (HTM), and next detail each component in our method. We propose a zero-shot learning solution for automatically building the planning graph using only a context vector of the new environment. Our idea is that, after seeing many different environments and corresponding states of the system during training, given a new environment we should be able to effectively hallucinate possible system states. We can then use these hallucinations in lieu of real samples from the system in order to build the planning graph. To generate images conditioned on a context, we implement a CVAE as depicted in Figure 1. During training, we learn the prior latent distribution r ψ (z|c), modeled as a Gaussian with mean µ(c) and covariance matrix Σ(c), where µ(·) and Σ(·) are learned non-linear neural network transformations. During testing, when prompted with a new context vector c, we can sample latent vectors z 1,..., z N | c ∼ N (µ(c), Σ(c)), and pass them through the decoder p θ (x|z, c) for hallucinating samples in replacement of exploration data. A critical component in the SPTM method is the connectivity classifier that decides which image transitions are feasible. False positives may in impossible short-cuts in the graph, while false negatives can make the plan unnecessarily long. In, the classifier was trained discriminatively, using observations in the data that were reached within h steps as positive examples, and more than l steps as negative examples, where h and l are chosen arbitrarily. In practice, this leads to three important problems. First, this method is known to be sensitive to the choice of positive and negative labeling. Second, training data are required to be long, non-cyclic trajectories for a high likelihood of sampling'true' negative samples. However, self-supervised interaction data often resembles random walks that repeatedly visit a similar state, leading to inconsistent estimates on what constitutes negative data. Third, since the classifier is only trained to predict positively for temporally nearby images and negatively for temporally far away images, its predictions of medium-distance images can be arbitrary. This creates both false positives and false negatives, thereby increasing shortcuts and missing edges in the graph. To solve these problems, we propose to learn a connectivity score using contrastive predictive loss. Similar to CVAE, we initialize a CPC encoder g enc that takes in both observation and context, and a density-ratio model f k that does not depend on the context. Through optimizing the CPC objective, f k of positive pairs are encouraged to be distinguishable from that of negative pairs. Thus, it serves as a proxy for the temporal distance between two observations, leading to a connectivity score for planning. Theoretically, CPC loss is better motivated than the classification loss in SPTM as it structures the latent space on a clear objective: maximize the mutual information between current and future observations. In practice, this in less hyperparameter tuning and a smoother distance manifold in the representation space. Finally, instead of only sampling from the same trajectory as done in SPTM, our negative data are collected by sampling from the latent space of a trained CVAE or the replay buffer. Without this trick, we found that the SPTM classifier fails to handle self-supervised data. l) ). This score reflects the difficulty in transitioning to the next state from the current state by self-supervised exploration. The learned connectivity graph G can be viewed as a topological memory upon which we can use conventional graph planning methods to efficiently perform visual planning. In the third step, we find the shortest path using Dijkstra's algorithm on the learned connectivity graph G between the start and end node. In the fourth step, we apply our policy to follow the visual plan, reaching the next node in our shortest path and replan every fixed number of steps until we reachô goal. For the policy, we train an inverse model which predicts actions given two observations that are within h steps apart. Maximum likelihood trajectory with Dijkstra's. We show that the CPC loss can be utilized to cast the planning problem as an inference problem, and in an effective planning algorithm. After training the CPC objective to convergence, we have To estimate p(o t+k |o t)/p(o t+k), we compute the normalizing factor o ∈V [f k (o, o t)] for each o t by averaging over all nodes in the graph. Let's define our non-negative weight from o t to o t+k as A shortest-path planning algorithm finds T, o 0,..., o T that minimizes, Thus, assuming that the self-supervised data distribution is approximately uniform, the shortest path algorithm with proposed weight ω maximizes a lower bound on the trajectory likelihood given the start and goal states. In practice, this leads to a more stable planning approach and yields more feasible plans. Reinforcement Learning. Most of the study of data-driven planning has been under the model-free RL framework. However, the need to design a reward function, and the fact that the learned policy does not generalize to tasks that are not defined by the specific reward, has motivated the study of model-based approaches. Recently, investigated model-based RL from pixels on Mujoco and Atari domains, but did not study generalization to a new environment. explored model-based RL with image-based goals using visual model predictive control (visual MPC). These methods rely on video prediction, and are limited in the planning horizon due to accumulating errors. In comparison, our method does not predict full trajectories but only individual images, mitigating this problem. Our method can also use visual MPC as a replacement for the visual servoing policy. Self-supervised learning. Several studies investigated planning goal directed behavior from data obtained offline, e.g., by self-supervised robot interaction. Nair et al. used an inverse model to reach local sub-goals, but require human demonstrations of long-horizon plans. Wang et al. solve the visual planning problem using a conditional version of Causal InfoGAN. However, as training GAN is unstable and requires tedious model selection, we opted for the CVAE-based approach, which is much more robust. Classical planning and representation learning. In classical planning literature, task and motion planning also separates the high-level planning and the low-level controller. In these works, domain knowledge is required to specify preconditions and effects at the task level. Our approach only requires data collected through self-supervised interaction. Other studies that bridge between classical planning and representation learning include. These works, however, do not consider zero-shot generalization. While Srinivas et al. and Qureshi et al. learn representations that allow goal-directed planning to unseen environments, they require expert training trajectories. Ichter and Pavone also generalizes motion planning to new environments, but require a collision checker and valid samples from test environments. Recent work in visual planning (e.g., ) focused on real robotic tasks with visual input. While impressive, such can be difficult to reproduce or compare. For example, it is not clear whether manipulating a rope with the PR2 robot is more or less difficult than manipulating a rigid object among many visual distractors. In light of this difficulty, we propose a suite of simulated tasks with an explicit difficulty scale and clear evaluation metrics. Our domains consider moving a rigid object between obstacles using Mujoco, and by varying the obstacle positions, we can control the planning difficulty. For example, placing the object in a cul-de-sac would require non-trivial planning compared to simply moving around an obstacle along the way to the goal. We thus create two domains, as seen in Figure 2: 1. Block wall:: A green block navigates around a static red obstacle, which can vary in position. 2. Block wall with complex obstacle: Similar to the above, but here the wall is a 3-link object which can vary in position, joint angles, and length, making the task significantly harder. With these domains, we aim to asses the following attributes: • Does HTM improve visual plan quality over state-of-the-art VP methods? • How does HTM execution success rate compare to state-of-the-art VP methods? • How well does HTM generalize its planning to unseen contexts? We discuss our evaluation metrics for these attributes in Section 5.1. To fully assess success of HTM relative to other state-of-the-art VP methods, we run these evaluation metrics on SPTM and Visual Foresight. In the first baseline, since vanilla SPTM cannot plan in a new environment, we use the same samples generated by the same CVAE as HTM, and then build the graph by assigning edge weights in the graph proportional to their exponentiated SPTM classifier score. 3 We also give it the same negative sampling proceedure as HTM. The same low-level controller is also used to follow the plans. In the second baseline, Visual Foresight trains a video prediction model, and then performs model predictive control (MPC) which finds the optimal action sequence through random shooting. For the random shooting, we used 3 iterations of the cross-entropy method with 200 sample sequences. The MPC acts for 10 steps and replans, where the planning horizon T is 15. We use the state-of-the-art video predictor as proposed by Lee et al. and the public code provided by the authors. For evaluating trajectories in random shooting, we studied two cost functions that are suitable for our domains: pixel MSE loss and green pixel distance. The pixel MSE loss computes the pixel distance between the predicted observations and the goal image. This provides a sparse signal when the object pixels in the plan can overlap with those of the goal. We also investigate a cost function that uses prior knowledge about the task -the position of the moving green block, which is approximated by calculating the center of mass of the green pixels. As opposed to pixel MSE, the green pixel distance provides a smooth cost function which estimates the normalized distance between the estimated block positions of the predicted observations and the goal image. Note that this assumes additional domain knowledge compared to HTM. We design a set of tests that measure both qualitative and quantitative performance of an algorithm. To motivate the need for qualitative metrics, we reiterate the importance of planning interpretability; it is highly desirable that the generated plan visually make sense so as to allow a human to approve of the plan prior to execution. Qualitative Visual plans have the essential property of being intuitive, in that the imagined trajectory is perceptually sensible. Since these qualities are highly subjective, we devised a set of tests to evaluate plans based on human visual perception. For each domain, we asked 5 participants to visually score 5 randomly generated plans from each model by answering the following questions: Fidelity: Does the pixel quality of the images resemble the training data?; Feasibility: Is each transition in the generated plan executable by a single action step?; and Completeness: Is the goal reachable from the last image in the plan using a single action? Answers were in the range, where 0 denotes No to the proposed question and 1 means Yes. The mean opinion score were calculated for each model. Quantitative In addition to generating visually sensible trajectories, a planning algorithm must also be able to successfully navigate towards a predefined goal. Thus, for each domain, we selected 20 start and goal images, each with an obstacle configuration unseen during training. Success was measured by the ability to get within some L2 distance to the goal in a n steps or less, where the distance threshold and n varied on the domain but was held constant across all models. A controller specified by the algorithm executed actions given an imagined trajectory, and replanning occurred every r steps. Specific details can be found in the Appendix D. As shown in Table 5.2, HTM outperforms all baselines in both qualitative and quantitative measurements across all domains. In the simpler block wall domain, Visual Foresight with green pixel distance only succeeds under the assumption of additional state information of the object's location. the other algorithms do not have. However, in the complex obstacle domain, Visual Foresight fails to perform comparably to our algorithm, regardless of the additional assumption. We also compared our method with SPTM, using the same inverse model and CVAE to imagine testing samples. However, without a robust classification loss and improved method of weighting the graph's edges, SPTM often fails to find meaningful transitions. In regards to perceptual evaluation, Visual Foresight generates realistic transitions, as seen by the high participant scores for feasibility. However, the algorithm is limited in creating a visual plan within the optimal T = 15 timesteps. 4 Thus, when confronted with a challenging task of navigating around a convex shape where the number of timesteps required exceeds T, Visual Foresight fails to construct a reliable plan (see Figure 3), and thus lacks plan completeness. Conversely, SPTM is able to imagine some trajectory that will reach the goal state. However, as mentioned above and was confirmed in the perceptual scores, SPTM fails to select feasible transitions, such as imagining a trajectory where the block will jump across the wall or split into two blocks. Our approach, on the other hand, received the highest scores of fidelity, feasibility, and completeness. Finally, we show in Figure 3 the of our two proposed improvements to SPTM in isolation. The clearly show that a classifier using contrastive loss outperforms that which uses Binary Cross Entropy (BCE) loss, and furthermore that the inverse of the score function for edge weighting is more successful than the best tuned version of binary edge weights. Table 1: Qualitative and quantitative evaluation for the the block wall and block wall with complex obstacle domains. Qualitative data also displays the 95% confidence interval. Note HTM refers to edge weighting using the energy model, and is weighting using the density ratio, as described in 3.3. For the score function, we denote the energy model structured with contrastive loss as CPC and the classifier as proposed in with BCE loss as SPTM. For the edge weighting function, we test the binary edge weighting from the original SPTM paper, the inverse of the score function, and the inverse of the normalized score function. We propose a method that is visually interpretable and modular -we first hallucinate possible configurations, then compute a connectivity between them, and then plan. Our HTM can generalize to unseen environments and improve visual plan quality and execution success rate over state-of-the-art VP methods. Our suggest that combining classical planning methods with data-driven perception can be helpful for long-horizon visual planning problems, and takes another step in bridging the gap between learning and planning. In future work, we plan to combine HTM with Visual MPC for handling more complex objects, and use object-oriented planning for handling multiple objects. Another interesting aspect is to improve planning by hallucinating samples conditioned on the start and goal configurations, which can help reduce the search space during planning. In this section, we assume the dataset as described in VPA, D = {o . There are two ways of learning a model to distinguish the positive from the negative transitions. Classifier: As noted above, SPTM first trains a classifier which distinguishes between an image pair that is within h steps apart, and the images that are far apart using random sampling. The classifier is used to localize the current image and find possible next images for planning. In essence, the classifier contains the encoder g θ that embeds the observation x and the the score function f that takes the embedding of each image and output the logit for a sigmoid function. The binary cross entropy loss of the classifier can be written as follows:, where z − t is a random sample from D. Energy model: Another form of discriminating the the positive transition out of negative transitions is through an energy model. Oord et al. learn the embeddings of the current states that are predictive of the future states. Let g be an encoder of the input x and z = g θ (x) be the embedding. The loss function can be described as a cross entropy loss of predicting the correct sample from N + 1 samples which contain 1 positive sample and N negative samples:, where f ψ (u, v) = exp (u T ψv) and z Figure 6: Sample observations (top) and contexts (bottom). In this domain, an object can be translated and rotated (SE) slightly per timestep. The data are collected from 360 different object shapes with different number of building blocks between 3 to 7. Each object is randomly initialized 50 times and each episode has length 30. The goal is to plan a manipulation of an unseen object through the narrow gap between obstacles in zero-shot.

We propose Hallucinative Topological Memory (HTM), a visual planning algorithm that can perform zero-shot long horizon planning in new environments.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose an end-to-end differentiable planning network for graphs. This can be applicable to many motion planning problems


Question: Represent the scientific article to find a short TLDR

Question: Deep Neural Networks (DNNs) thrive in recent years in which Batch Normalization (BN) plays an indispensable role. However, it has been observed that BN is costly due to the reduction operations. In this paper, we propose alleviating the BN’s cost by using only a small fraction of data for mean & variance estimation at each iteration. The key challenge to reach this goal is how to achieve a satisfactory balance between normalization effectiveness and execution efficiency. We identify that the effectiveness expects less data correlation while the efficiency expects regular execution pattern. To this end, we propose two categories of approach: sampling or creating few uncorrelated data for statistics’ estimation with certain strategy constraints. The former includes “Batch Sampling (BS)” that randomly selects few samples from each batch and “Feature Sampling (FS)” that randomly selects a small patch from each feature map of all samples, and the latter is “Virtual Dataset Normalization (VDN)” that generates few synthetic random samples. Accordingly, multi-way strategies are designed to reduce the data correlation for accurate estimation and optimize the execution pattern for running acceleration in the meantime. All the proposed methods are comprehensively evaluated on various DNN models, where an overall training speedup by up to 21.7% on modern GPUs can be practically achieved without the support of any specialized libraries, and the loss of model accuracy and convergence rate are negligible. Furthermore, our methods demonstrate powerful performance when solving the well-known “micro-batch normalization” problem in the case of tiny batch size. Recent years, Deep Neural Networks (DNNs) have achieved remarkable success in a wide spectrum of domains such as computer vision BID16 and language modeling BID4. The success of DNNs largely relies on the capability of presentation benefit from the deep structure BID5. However, training a deep network is so difficult to converge that batch normalization (BN) has been proposed to solve it BID14. BN leverages the statistics (mean & variance) of mini-batches to standardize the activations. It allows the network to go deeper without significant gradient explosion or vanishing BID23 BID14. Moreover, previous work has demonstrated that BN enables the use of higher learning rate and less awareness on the initialization BID14, as well as produces mutual information across samples BID21 or introduces estimation noises BID2 for better generalization. Despite BN's effectiveness, it is observed that BN introduces considerable training overhead due to the costly reduction operations. The use of BN can lower the overall training speed (mini second per image) by >45%, especially in deep models. To alleviate this problem, several methods were reported. Range Batch Normalization (RBN) BID1 accelerated the forward pass by estimating the variance according to the data range of activations within each batch. A similar approach, L 1 -norm BN (L1BN), simplified both the forward and backward passes by replacing the L 2 -norm variance with its L 1 -norm version and re-derived the gradients for backpropagation (BP) training. Different from the above two methods, Self-normalization BID15 provided another solution which totally eliminates the need of BN operation with an elaborate activation function called "scaled exponential linear unit" (SELU). SELU can automatically force the activation towards zero mean and unit variance for better convergence. Nevertheless, all of these methods are not sufficiently effective. The strengths of L1BN & RBN are very limited since GPU has sufficient resources to optimize the execution speed of complex arithmetic operations such as root for the vanilla calculation of L 2 -norm variance. Since the derivation of SELU is based on the plain convolutional network, currently it cannot handle other modern structures with skip paths like ResNet and DenseNet. In this paper, we propose mitigating BN's computational cost by just using few data to estimate the mean and variance at each iteration. Whereas, the key challenge of this way lies at how to preserve the normalization effectiveness of the vanilla BN and improve the execution efficiency in the meantime, i.e. balance the effectiveness-efficiency trade-off. We identify that the effectiveness preservation expects less data correlation and the efficiency improvement expects regular execution pattern. This observation motivates us to propose two categories of approach to achieve the goal of effective and efficient BN: sampling or creating few uncorrelated data for statistics' estimation with certain strategy constraints. Sampling data includes "Batch Sampling (BS)" that randomly selects few samples from each batch and "Feature Sampling (FS)" that randomly selects a small patch from each feature map (FM) of all samples; creating data means "Virtual Dataset Normalization (VDN)" that generates few synthetic random samples, inspired by BID22. Consequently, multi-way strategies including intra-layer regularity, inter-layer randomness, and static execution graph during each epoch, are designed to reduce the data correlation for accurate estimation and optimize the execution pattern for running acceleration in the meantime. All the proposed approaches with single-use or joint-use are comprehensively evaluated on various DNN models, where the loss of model accuracy and convergence rate is negligible. We practically achieve an overall training speedup by up to 21.7% on modern GPUs. Note that any support of specialized libraries is not needed in our work, which is not like the network pruning BID32 or quantization BID12 requiring extra library for sparse or low-precision computation, respectively. Most previous acceleration works targeted inference which remained the training inefficient BID26 BID20 BID19 BID31 BID9, and the rest works for training acceleration were orthogonal to our approach BID7 BID29. Additionally, our methods further shows powerful performance when solving the well-known "micro-batch normalization" problem in the case of tiny batch sizes. In summary, the major contributions of this work are summarized as follows.• We propose a new way to alleviate BN's computational cost by using few data to estimate the mean and variance, in which we identify that the key challenge is to balance the normalization effectiveness via less data correlation and execution efficiency via regular execution pattern.• We propose two categories of approach to achieve the above goal: sampling (BS/FS) or creating (VDN) few uncorrelated data for statistics' estimation, in which multi-way strategies are designed to reduce the data correlation for accurate estimation and optimize the execution pattern for running acceleration in the meantime. The approaches can be used alone or jointly.• Various benchmarks are evaluated, on which up to 21.7% practical acceleration is achieved for overall training on modern GPUs with negligible accuracy loss and without specialized library support.• Our methods are also extended to the micro-BN problem and achieve advanced performance 1.In order to make this paper easier for understanding, we present the organization of the whole paper in FIG0 The activations in one layer for normalization can be described by a d-dimensional activation feature DISPLAYFORM0, where for each feature we have DISPLAYFORM1 Note that in convolutional (Conv) layer, d is the number of FMs and m equals to the number of points in each FM across all the samples in one batch; while in fully-connected (FC) layer, d and m are the neuron number and batch size, respectively. BN uses the statistics (mean E[ DISPLAYFORM2 of the intra-batch data for each feature to normalize activation by DISPLAYFORM3 where DISPLAYFORM4 are trainable parameters introduced to recover the representation capability, is a small constant to avoid numerical error, and DISPLAYFORM5 The detailed operations of a BN layer in the backward pass can be found in Appendix C. DISPLAYFORM6 Iter. per second. TAB3 ; (b) usual optimization of the reduction operation using adder tree; (c) the computational graph of BN in the forward pass (upper) and backward pass (lower); (d) the computation graph of BN using few data for statistics' estimation in forward pass (upper) and backward pass (lower). x is neuronal activations, µ and σ denote the mean and standard deviation of x within one batch, respectively, and is the summation operation. From FIG0, we can see that adding BN will significantly slow down the training speed (iterations per second) by 32%-43% on ImageNet. The reason why BN is costly is that it contains several "reduction operations", i.e. m j=1. We offer more thorough data analysis in Appendix E. If the reduction operations are not optimized, it's computational complexity should be O(m). With the optimized parallel algorithm proposed in BID3, the reduction operation is transformed to cascaded adders of depth of log(m) as shown in FIG0. However, the computational cost is still high since we usually have m larger than one million. As shown in FIG0, the red " "s represent operations that contain summations, which cause the BN inefficiency. Motivated by the above analysis, decreasing the effective value of m at each time for statistics estimation seems a promising way to reduce the BN cost for achieving acceleration. To this end, we propose using few data to estimate the mean and variance at each iteration. For example, if m changes to a much smaller value of s, equation FORMULA5 can be modified as DISPLAYFORM0 Under review as a conference paper at ICLR 2019 where x (k) s denotes the small fraction of data, s is the actual number of data points, and we usually have s m. Here we denote s/m as Sampling Ratio (it includes both the cases of sampling and creating few data in Section 3.1 and 3.2, respectively). Since the reduction operations in the backward pass can be parallelized whereas in the forward pass, the variance can not be calculated until mean is provided (which makes it nearly twice as slow as backward pass), we just use few data in the forward pass. The computational graph of BN using few data is illustrated in FIG0. The key is how to estimate E[DISPLAYFORM1 for each neuron or FM within one batch with much fewer data. The influence on the backward pass is discussed in Appendix C. Although using few data can reduce the BN's cost dramatically, it will meet an intractable challenge: how to simultaneously preserve normalization effectiveness and improve the execution efficiency. On one side, using few data to estimate the statistics will increase the estimation error. By regarding the layers with high estimation error as unnormalized ones, we did contrast test in FIG1 . The mean & variance will be scaled up exponentially as network deepens, which causes the degradation of BN's effectiveness. This degradation can be recovered from two aspects.• Intra-layer. For the reason that the estimation error is not only determined by the amount of data but also the correlation between them, we can sample less correlated data within each layer to improve the estimation accuracy.• Inter-layer. As depicted in by FIG1, the intermittent BN configuration (i.e. discontinuously adding BN in different layers) can also prevent the statistics scaling up across layers. This motivates us that as long as layers with high estimation error are discontinuous, the statistics shift can still be constrained to a smaller range. Therefore, to reduce the correlation between estimation errors in different layers can also be beneficial to improve the accuracy of the entire model, which can be achieved by sampling less correlated data between layers. On the other side, less data correlation indicates more randomness which usually causes irregular memory access irregular degrading the running efficiency. In this paper, we recognize that the overhead of sampling can be well reduced by using regular and static execution patterns, which is demonstrated with ablation study at Fig In a nutshell, careful designs are needed to balance the normalization effectiveness via less data correlation and the execution efficiency via regular execution pattern. Only in this way, it is possible to achieve practical acceleration with little accuracy loss, which is our major target in this work. Based on the above analysis, we summarize the design considerations as follows.• Using few data for statistics' estimation can effectively reduce the computational cost of BN operations. Whereas, the effectiveness-efficiency trade-off should be well balanced.• Less data correlation is promising to reduce the estimation error and then guarantees the normalization effectiveness.• More regular execution pattern is expected for efficient running on practical platforms. To reach the aforementioned goal, we propose two categories of approach in this section: sampling or creating few uncorrelated data for statistics' estimation. Furthermore, multi-way strategies to balance the data correlation and execution regularity. Here "sampling" means to sample a small fraction of data from the activations at each layer for statistics' estimation. As discussed in the previous section, mining uncorrelated data within and between layers is critical to the success of sampling-based BN. However, the correlation property of activations in deep networks is complex and may vary across network structures and datasets. Instead, we apply a hypothesis-testing approach in this work. We first make two empirical assumptions:• Hypothesis 1. Within each layer, data belonging to the different samples are more likely to be uncorrelated than those within the same sample.• Hypothesis 2. Between layers, data belonging to different locations and different samples are less likely to be correlated. Here "location" means coordinate within FMs. These two assumptions are based on the basic nature of real-world data and the networks, thus they are likely to hold in most situations. They are further evaluated through experiments in Section 4.1.Based on above hypotheses, we propose two uncorrelated-sampling strategies: Batch Sampling (BS) and Feature Sampling (FS). The detailed Algorithms can be found in Alg. 1 and 2, respectively, Appendix A.• BS FIG2 ) randomly selects few samples from each batch for statistics' estimation. To reduce the inter-layer data correlation, it selects different samples across layers following Hypothesis 2.• FS FIG2 ) randomly selects a small patch from each FM of all samples for statistics' estimation. Since the sampled data come from all the samples thus it has lower correlation within each layer following Hypothesis 1. Furthermore, it samples different patch locations across layers to reduce the inter-layer data correlation following Hypothesis 2.A Naive Sampling (NS) is additionally proposed as a comparison baseline, as shown in FIG2. NS is similar to BS while the sampling index is fixed across layers, i.e. consistently samples first few samples within each batch. Regular and Static Sampling. In order to achieve practical acceleration on GPU, we expect more regular sampling pattern and more static sampling index. Therefore, to balance the estimation effectiveness and execution efficiency, we carefully design the following sampling rules: In BS, the selected samples are continuous and the sample index for different channels are shared, while they are independent across layers; In FS, the patch shape is rectangular and the patch location is shared by different channels and samples within each layer but variable as layer changes. Furthermore, all the random indexes are updated only once for each epoch, which guarantees a static computational graph during the entire epoch. Instead of sampling uncorrelated data, another plausible solution is to directly create uncorrelated data for statistics' estimation. We propose Virtual Dataset Normalization (VDN) 2 to implement it, as illustrated in FIG2. VDN is realized with the following three steps: calculating the statistics of the whole training dataset offline; generating s virtual samples 3 at each iteration to concatenate with the original real inputs as the final network inputs. using data from only virtual samples at each layer for statistics' estimation. Due to the independent property of the synthesized data, they are more uncorrelated than real samples thus VDN can produce much more accurate estimation. The detailed implementation algorithm can be found in Alg. 3, Appendix A. The sampling approach and the creating approach can be used either in a single way or in a joint way. Besides the single use, a joint use can be described as DISPLAYFORM0 where x s denotes the sampled real data while x v represents the created virtual data. β is a controlling variable, which indicates how large the sampled data occupy the whole data for statistics' estimation within each batch: when β = 0 or 1, the statistics come from single approach (VDN or any sampling strategy); when β ∈, the final statistics are a joint value as shown in equation 4. A comparison between different using ways is presented in TAB2, where "d.s." denotes "different samples"; "d.l." stands for "different locations", and "g.i." indicates "generated independent data". Compared to NS, BS reduces the inter-layer correlation via selecting different samples across layers; FS reduces both the intra-layer and inter-layer correlation via using data from all samples within each layer and selecting different locations across layers, respectively. Though VDN has a similar inter-layer correlation with NS, it slims the intra-layer correlation with strong data independence. A combination of BS/FS and VDN can inherit the strength of different approaches, thus achieve much lower accuracy loss. Experimental Setup. All of our proposed approaches are validated on image classification task using CIFAR-10, CIFAR-100 and ImageNet datasets from two perspectives: effectiveness evaluation and efficiency execution. To demonstrate the scalability and generality of our approaches on deep networks, we select ResNet-56 on CIFAR-10 & CIFAR-100 and select ResNet-18 and DenseNet-121 on ImageNet 4. The model configuration can be found in TAB3. The means and variances for BN are locally calculated in each GPU without inter-GPU synchronization as usual. We denote our approaches as the format of "Approach-Sampled size/Original size-sampling ratio(%)". For instance, if we assume batch size is 128, "BS-4/128-3.1%" denotes only 4 samples are sampled in BS and the sampling ratio equals to 4 128 = 3.1%. Similarly, "FS-1/32-3.1%" implies a 1 32 = 3.1% patch is sampled from each FM, and "VDN-1/128-0.8%" indicates only one virtual sample is added. The traditional BN is denoted as "BN-128/128-100.0%". Other experimental configurations can be found in Appendix B. Convergence Analysis. FIG3 shows the top-1 validation accuracy and confidential interval of ResNet-56 on CIFAR-10 and CIFAR-100. On one side, all of our approaches can well approximate the accuracy of normal BN when the sampling ratio is larger than 2%, which evidence their effectiveness. On the other side, all the proposed approaches perform better than the NS baseline. In particular, FS performs best, which is robust to the sampling ratio with negligible accuracy loss (e.g. at sampling ratio=1.6%, the accuracy degradation is -0.087% on CIFAR-10 and +0.396% on CIFAR-100). VDN outperforms BS and NS with a large margin in extremely small sampling ratio (e.g. 0.8%), whereas the increase of virtual batch size leads to little improvement on accuracy. BS is constantly better than NS. Furthermore, an interesting observation is that the BN sampling could even achieve better accuracy sometimes, such as NS-8/128(72.6±1.5%), BS-8/128(72.3±1.5%), and FS-1/64(71.2±0.96%) against the baseline (70.8±1%) on CIFAR-100. FIG4 further shows the training curves of ResNet-56 on CIFAR-10 under different approaches. It reveals that FS and VDN would not harm the convergence rate, while BS and NS begin to degrade the convergence when the sampling ratio is smaller than 1.6% and 3.1%, respectively. TAB4 shows the top-1 validation error on ImageNet under different approaches. With the same sampling ratio, all the proposed approaches significantly outperform NS, and FS surpasses VDN and BS. Under the extreme sampling ratio of 0.78%, NS and BS don't converge. Due to the limitation of FM size, the smallest sampling ratio of FS is 1.6%, which has only -0.5% accuracy loss. VDN can still achieve relatively low accuracy loss (1.4%) even if the sampling ratio decreases to 0.78%. This implies that VDN is effective for normalization. Moreover, by combining FS-1/64 and VDN-2/128, we get the lowest accuracy loss (-0.2%). This further indicates that VDN can be combined with other sampling strategies to achieve better . Since training DenseNet-121 is time-consuming, we just report the with FS/BS-VDN joint use. Although DenseNet-121 is more challenging than ResNet-18 due to the much deeper structure, the "FS-1/64 + VDN-2/64" can still achieve very low accuracy loss (-0.6%). "BS-1/64 + VDN-2/64" has a little higher accuracy loss, whereas it still achieves better than NS. In fact, we observed gradient explosion if we just use VDN on very deep network (i.e. DenseNet-121), which can be conquered through jointly applying VDN and other proposed sampling approach (e.g. FS+VDN). FIG5 illustrates the training curves for better visualization of the convergence. Except for the BS with extremely small sampling ratio (0.8%) and NS, other approaches and configurations can achieve satisfactory convergence. Here, we further evaluate the fully random sampling (FRS) strategy, which samples completely random points in both the batch and FM dimensions. We can see that FRS is less stable compared with our proposed approaches (except the NS baseline) and achieves much lower accuracy. One possible reason is that under low sampling ratio, the sampled data may occasionally fall into the worse points, which lead to inaccurate estimation of the statistics. Correlation Analysis. In this section, we bring more empirical analysis to the data correlation that affects the error of statistical estimation. Here we denote the estimation errors at l th layer as E DISPLAYFORM0 s are the estimated mean & variance from the sampled data (including the created data in VDN) while µ (l) & σ (l) are the ground truth from the vanilla BN for the whole batch. The analysis is conducted on ResNet-56 over CIFAR-10. The estimation errors of all layers are recorded throughout the first training epoch. FIG6 and FIG7 present the distribution of estimation errors for all layers and the inter-layer correlation between estimation errors, respectively. In FIG6, FS demonstrates the least estimation error within each layer which implies its better convergence. The estimation error of VDN seems similar to BS and NS here, but we should note that it uses much lower sampling ratio of 0.8% compared to others of 3.1%. FIG7, it can be seen that BS presents obviously less inter-layer correlation than NS, which is consistent with previous experimental that BS can converge better than NS even though they have similar estimation error as shown in FIG6. For FS and VDN, although it looks like they present averagely higher correlations, there exist negative corrections which effectively improve the model accuracy. Moreover, FS produces better accuracy than NS and BS since its selected data come from all the samples with less correlation. After the normalization effectiveness evaluation, we will evaluate the execution efficiency which is the primary motivation. FIG8 shows the BN speedup during training and overall training improvement. In general, BS can gain higher acceleration ratio because it doesn't incur the fine-grained sampling within FMs like in FS and it doesn't require the additional calculation and concatenation of the virtual samples like in VDN. As for FS, it fails to achieve speedup on CIFAR-10 due to the small image size that makes the reduction of operations unable to cover the sampling overhead. The proposed approaches can obtain up to 2x BN acceleration and 21.8% overall training acceleration. TAB5 gives more on ResNet-18 using single approach and DenseNet-121 using joint approach. On ResNet-18, we perform much faster training compared with two recent methods for BN simplification BID1. On ResNet-18 we can achieve up to 16.5% overall training speedup under BS; on very deep networks with more BN layers, such as DenseNet-121, the speedup is more significant that reaches 23.8% under "BS+VDN" joint approach. "FS+VDN" is a little bit slower than "BS+VDN" since the latter one has a more regular execution pattern as aforementioned. Nonetheless, on a very deep model, we still recommend the "FS+VDN" version because it can preserve the accuracy better. The relationship be-tween sampling ratio and overall training speedup is represented in FIG0 which illustrates that 1) BS & FS can still achieve considerable speedup with a moderate sampling ratio; 2) BS can achieve more significant acceleration than FS, for its more regular execution pattern. It's worth noting that, our training speedup is practically obtained on modern GPUs without the support of specialized library that makes it easy-to-use. FIG8 (c) reveals that the regular execution pattern can significantly help us achieve practical speedup. 128/128 5.23 +0.38 RBN BID1 128 BN has been applied in most state-of-art DNN models BID8 BID24 since it was proposed. As aforementioned, BN standardizes the activation distribution to reduce the internal covariate shift. Models with BN have been demonstrated to converge faster and generalize better BID14 BID21. Recently, a model called Decorrelated Batch Normalization (DBN) was introduced which not only standardizes but also whitens the activations with ZCA whitening BID11. Although DBN further improves the normalization performance, it introduces significant extra computational cost. Simplifying BN has been proposed to reduce BN's computational complexity. For example, L1BN and RBN BID1 replace the original L 2 -norm variance with an L 1 -norm version and the range of activation values, respectively. From another perspective, Selfnormalization uses the customized activation function (SELU) to automatically shift activation's distribution BID15. However, as mentioned in Introduction, all of these methods fail to obtain a satisfactory balance between the effective normalization and computational cost, especially on large-scale modern models and datasets. Our work attempts to address this issue. Motivated by the importance but high cost of BN layer, we propose using few data to estimate the mean and variance for training acceleration. The key challenge towards this goal is how to balance the normalization effectiveness with much less data for statistics' estimation and the execution efficiency with irregular memory access. To this end, we propose two categories of approach: sampling (BS/FS) or creating (VDN) few uncorrelated data, which can be used alone or jointly. Specifically, BS randomly selects few samples from each batch, FS randomly selects a small patch from each FM of all samples, and VDN generates few synthetic random samples. Then, multi-way strategies including intra-layer regularity, inter-layer randomness, and static execution graph are designed to reduce the data correlation and optimize the execution pattern in the meantime. Comprehensive experiments evidence that the proposed approaches can achieve up to 21.7% overall training acceleration with negligible accuracy loss. In addition, VDN can also be applied to the micro-BN scenario with advanced performance. This paper preliminary proves the effectiveness and efficiency of BN using few data for statistics' estimation. We emphasize that the training speedup is practically achieved on modern GPUs, and we do not need any support of specialized libraries making it easy-to-use. Developing specialized kernel optimization deserves further investigation for more aggressive execution benefits. Notations. We use the Conv layer for illustration, which occupies the major part of most modern networks BID8 BID10. The batched features can be viewed as a 4D tensor. We use "E 0,1,2 " and "V ar 0,1,2 " to represent the operations that calculate the means and variances, respectively, where "0, 1, 2" denotes the dimensions for reduction. Data: input batch at layer l: DISPLAYFORM0 for ep ∈ all epochs do for l ∈ all layers do if BS: begin l = randint(0, N − n s); else NS: DISPLAYFORM1 Algorithm 2: FS Algorithm Data: input batch at layer l: DISPLAYFORM2 for ep ∈ all epochs do for l ∈ all layers do begin DISPLAYFORM3 All the experiments on CIFAR-10 & CIFAR-100 are conducted on a single Nvidia Titan Xp GPU. We use a weight decay of 0.0002 for all weight layers and all models are trained by 130 epochs. The initial learning rate is set to 0.1 and it is decreased by 10x at 50, 80, 110 epoch. During training, we adopt the "random flip left & right" and all the input images are randomly cropped to 32 × 32. Each model is trained from scratch for 5 times in order to reduce random variation. For ImageNet, We use 2 Nvidia Tesla V100 GPUs on DGX station for ResNet-18 and 3 for DenseNet-121. We use a weight decay of 0.0001 for all weight layers and all models are trained by 100 epochs. The initial learning rate is set to 0.1/256×"Gradient batch size" and we decrease the learning rate by 10x at 30, 60, 80, 90 epoch. During training, all the input images are augmented by random flipping and cropped to 224 × 224. We evaluate the top-1 validation error on the validation set using the centered crop of each image. To reduce random variation, we use the average of last 5 epochs to represent its final error rate. Besides, Winograd BID17 ) is applied in all models to speedup training. Compute Cycle and Memory Access. Our proposed approaches can effectively speedup forward pass. Under the condition that we use s m data to estimation the statistics for each feature, the total accumulation operations are significantly reduced from m − 1 to s − 1. If using the adder tree optimization illustrated in Section 2.1, the tree's depth can be reduced from log(m) to log(s). Thus, the theoretical compute speedup for the forward pass can reach log s (m) times. For instance, if the FM size is 56 × 56 with batch size of 128 (m = 56 × 56 × 128), under sampling ratio of 1/32, the compute speedup will be 36.7%. The total memory access is reduced by m/s times. For example, when the sampling ratio is 1/32, only 3.1% data need to be visited. This also contributes a considerable part in the overall speedup. Speedup in the Backward Pass. The BN operations in the forward pass have been shown in equation FORMULA3 - FORMULA7. Based on the derivative chain rule, we can get the corresponding operations in the backward pass as follows DISPLAYFORM0 Figure 11: Influence of decay rate for moving average. The above equation reveals that an appropriately smaller α might scale down the estimation error, thus produces better validation accuracy. To verify this prediction, the experiments are conducted on ResNet-56 over CIFAR-10 and using BS-1/128(0.78%) sampling. As shown in FIG0, it's obvious that there exists a best decay rate setting (here is 0.7) whereas the popular decay rate is 0.9. The performance also decays when decay rate is smaller than 0.7, which is because a too small α may lose the capability to record the moving estimation, thus degrade the validation accuracy. This is interesting because the decay rate is usually ignored by researchers, but the default value is probably not the best setting in our context. Figure 12: BN network for profiling. To further demonstrate that reduction operations are the major bottleneck of BN, we build a pure BN network for profiling as shown in FIG0. The network's input is a variable with shape of which is initialized following a standard normal distribution. The network is trained for 100 iterations and the training time is recorded. We overwrite the original BN's codes and remove the reduction operations in both forward pass and backward pass for contrast test. We use three different GPUs: K80, Titan Xp, and Tesla V100 to improve the reliability. The are shown in TAB7. We can see that on all the three GPUs, reduction operations take up to >60% of the entire operation time in BN. As a , it's solid to argue that the reduction operations are the bottleneck of BN. Micro-BN aims to alleviate the diminishing of BN's effectiveness when the amount of data in each GPU node is too small to provide a reliable estimation of activation statistics. Previous work can be classified to two categories: Sync-BN BID30 and Local-BN BID0 BID28 BID13. The former addresses this problem by synchronizing the estimations from different GPUs at each layer, which induces significant inter-GPU data dependency and slows down training process. The latter solves this problem by either avoiding the use of batch dimension in the batched activation tensor for statistics or using additional information beyond current layer to calibrate the statistics. In terms of BN's efficiency preservation, we are facing a similar problem with micro-BN, thus our framework can be further extended to the micro-BN scenario. In Sync-BN: With FS, each GPU node executes the same patch sampling as normal FS; With BS, we can randomly select the statistics from a fraction of GPUs rather than all nodes; With VDN, the virtual samples can be fed into a single or few GPUs. The first one can just simplify the computational cost within each GPU, while the last two further optimize the inter-GPU data dependency. In Local-BN, since the available data for each GPU is already tiny, the BN sampling strategy will be invalid. Fortunately, the VDN can still be effective by feeding virtue samples into each node. Experiments. The normalization in Sync-BN is based on the statistics from multiple nodes through synchronization, which is equivalent to that in FIG5 with large batch size for each node. Therefore, to avoid repetition, here we just show the on Local-BN with VDN optimization. We let the overall batch size of 256 breaks down to 64 workers (each one has only 4 samples for local normalization). We use "(gradient batch size, statistics batch size)" of to denote the configuration. A baseline of with BN and one previous work Group Normalization (GN) BID28 are used for comparison. As shown in FIG0, although the reduction of batch size will degrade the model accuracy, our VDN can achieve slightly better (top-1 validation error rate: 30.88%) than GN (top-1 validation error rate: 30.96%), an advanced technique for this scenario with tiny batch size. This promises the correct training of very large model so that each single GPU node can only accommodate several samples.. FIG0: Illustration of the paper's organization. The Green and Purple markers (round circle with a star in the center) represent whether the effectiveness is preserved by reducing inter-layer or intralayer correlation (Green: inter-layer; Purple: intra-layer). Moreover, the consideration of regular & static execution pattern is applied to all approaches.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose the Warped Residual Network using a parallelizable warp operator for forward and backward propagation to distant layers that trains faster than the original residual neural network.


Question: Represent the scientific article to find a short TLDR

Question: Federated learning allows edge devices to collaboratively learn a shared model while keeping the training data on device, decoupling the ability to do model training from the need to store the data in the cloud. We propose Federated matched averaging (FedMA) algorithm designed for federated learning of modern neural network architectures e.g. convolutional neural networks (CNNs) and LSTMs. FedMA constructs the shared global model in a layer-wise manner by matching and averaging hidden elements (i.e. channels for convolution layers; hidden states for LSTM; neurons for fully connected layers) with similar feature extraction signatures. Our experiments indicate that FedMA outperforms popular state-of-the-art federated learning algorithms on deep CNN and LSTM architectures trained on real world datasets, while improving the communication efficiency. Edge devices such as mobile phones, sensor networks or vehicles have access to a wealth of data. However, due to concerns raised by data privacy, network bandwidth limitation, and device availability, it's unpractical to gather all local data to the data center and conduct centralized training. To address these concerns, federated learning is emerging (; ; ; ;) to allow local clients to collaboratively train a shared global model. The typical federated learning paradigm involves two stages: (i) clients train models over their datasets independently (ii) the data center uploads their locally trained models. The data center then aggregates the received models into a shared global model. One of the standard aggregation methods is FedAvg where parameters of local models are averaged element-wise with weights proportional to sizes of client datasets. FedProx adds a proximal term for client local cost functions, which limits the impact of local updates by restricting them to be close to the global model. Agnostic Federated Learning (AFL) , as another variant of FedAvg, optimizes a centralized distribution that is formed by a mixture of the client distributions. One shortcoming of the FedAvg algorithm is that coordinate-wise averaging of weights may have drastic detrimental effect on the performance and hence hinders the communication efficiency. This issue arises due to the permutation invariant nature of the neural network (NN) parameters, i.e. for any given NN there are many variations of it that only differ in the ordering of parameters and constitute local optima which are practically equivalent. Probabilistic Federated Neural Matching (PFNM) addresses this problem by finding permutation of the parameters of the NNs before averaging them. PFNM further utilizes Bayesian nonparametric machinery to adapt global model size to heterogeneity of the data. As a , PFNM has better performance and communication efficiency, however it was only developed for fully connected NNs and tested on simple architectures. Our contribution In this work (i) we demonstrate how PFNM can be applied to CNNs and LSTMs, however we find that it gives very minor improvement over weight averaging when applied to modern deep neural network architectures; (ii) we propose Federated Matched Averaging (FedMA), a new layers-wise federated learning algorithm for modern CNNs and LSTMs utilizing matching and model size adaptation underpinnings of PFNM; (iii) We empirically study FedMA with real datasets under the federated learning constraints. In this section we will discuss permutation invariance classes of prominent neural network architectures and establish the appropriate notion of averaging in the parameter space of NNs. We will begin with the simplest case of a single hidden layer fully connected network, moving on to deep architectures and, finally, convolutional and recurrent architectures. Permutation invariance of fully connected architectures A basic fully connected (FC) NN can be formulated asŷ = σ(xW 1)W 2 (without loss of generality, biases are omitted to simplify notation), where σ is the non-linearity (applied entry-wise). Expanding the preceding expression, where i· and ·i denote ith row and column correspondingly and L is the number of hidden units. Summation is a permutation invariant operation, hence for any {W 1, W 2} there are L! practically equivalent parametrizations if this basic NN. It is then more appropriate to writeŷ Recall that permutation matrix is an orthogonal matrix that acts on rows when applied on the left and on columns when applied on the right. Suppose {W 1, W 2} are optimal weights, then weights obtained from training on two homogeneous datasets X j, X j are It is now easy to see why naive averaging in the parameter space is not appropriate: with non-negligible probability Π j = Π j and (W 1 Π j + W 1 Π j)/2 = W 1 Π for any Π. To meaningfully average neural networks in the weight space we should first undo the permutation In this section we formulate practical notion of parameter averaging under the permutation invariance. Let w jl be lth neuron learned on dataset j (i.e. lth column of W Π j in the previous example), θ i denote the ith neuron in the global model, and c(·, ·) be an appropriate similarity function between a pair of neurons. Solution to the following optimization problem are the required inverse permutations: Then Π Solving matched averaging Objective function in equation 2 can be optimized using an iterative procedure: applying the Hungarian matching algorithm to find permutation {π j li} l,i corresponding to dataset j, holding other permutations {π j li} l,i,j =j fixed and iterating over the datasets. Important aspect of Federated Learning that we should consider here is the data heterogeneity. Every client will learn a collection of feature extractors, i.e. neural network weights, representing their individual data modality. As a consequence, feature extractors learned across clients may overlap only partially. To account for this we allow the size of the global model L to be an unknown variable satisfying max j L j ≤ L ≤ j L j where L j is the number of neurons learned from dataset j. That is, global model is at least as big as the largest of the local models and at most as big as the concatenation of all the local models. Next we show that matched averaging with adaptive global model size remains amendable to iterative Hungarian algorithm with a special cost. At each iteration, given current estimates of {π j li} l,i,j =j, we find a corresponding global model (this is typically a closed-form expression or a simple optimization sub-problem, e.g. a mean if c(·, ·) is Euclidean) and then we will use Hungarian algorithm to match this global model to neurons {w j l} L j l=1 of the dataset j to obtain a new global model with L ≤ L ≤ L + L j neurons. Due to data heterogeneity, local model j may have neurons not present in the global model built from other local models, therefore we want to avoid "poor" matches by saying that if the optimal match has cost larger than some threshold value, instead of matching we create a new global neuron from the corresponding local one. We also want a modest size global model and therefore penalize its size with some increasing function f (L). This intuition is formalized in the following extended maximum bipartite matching formulation: The size of the new global model is then L = max{i : π j li = 1, l = 1, . . ., L j}. We note some technical details: after the optimization is done, each corresponding Π T j is of size L j × L and is not a permutation matrix in a classical sense when L j = L. Its functionality is however similar: taking matrix product with a weight matrix W j Π T j implies permuting the weights to align with weights learned on the other datasets and padding with "dummy" neurons having zero weights (alternatively we can pad weights W j first and complete Π T j with missing rows to recover a proper permutation matrix). This "dummy" neurons should also be discounted when taking average. Without loss of generality, in the subsequent presentation we will ignore these technicalities to simplify the notation. To complete the matched averaging optimization procedure it remains to specify similarity c(·, ·), threshold and model size penalty f (·). Although one can consider application specific choices, here for simplicity we follow setup of. They arrived at a special case of equation 3 to compute maximum a posteriori estimate (MAP) of their Bayesian nonparametric model based on the Beta-Bernoulli process (BBP) , where similarity c(w jl, θ i) is the corresponding posterior probability of jth client neuron l generated from a Gaussian with mean θ i, and and f (·) are guided by the Indian Buffet Process prior . We refer to a procedure for solving equation 2 with the setup from as BBP-MAP. We note that their Probabilistic Federated Neural Matching (PFNM) is only applicable to fully connected architectures limiting its practicality. Our matched averaging perspective allows to formulate averaging of widely used architectures such as CNNs and LSTMs as instances of equation 2 and utilize the BBP-MAP as a solver. Before moving onto the convolutional and recurrent architectures, we discuss permutation invariance in deep fully connected networks and corresponding matched averaging approach. We will utilize this as a building block for handling LSTMs and CNN architectures such as VGG widely used in practice. We extend equation 1 to recursively define deep FC network: where n = 1,..., N is the layer index, Π 0 is identity indicating non-ambiguity in the ordering of input features x = x 0 and Π N is identity for the same in output classes. Conventionally σ(·) is any non-linearity except forŷ = x N where it is the identity function (or softmax if we want probabilities instead of logits). When N = 2, we recover a single hidden layer variant from equation 1. To perform matched averaging of deep FCs obtained from J clients we need to find inverse permutations for every layer of every client. Unfortunately, permutations within any consecutive pair of intermediate layers are coupled leading to a NP-hard combinatorial optimization problem. Instead we consider recursive (in layers) matched averaging formulation. Suppose we have {Π j,n−1}, then plugging {Π T j,n−1 W j,n} into equation 2 we find {Π j,n} and move onto next layer. The recursion base for this procedure is {Π j,0}, which we know is an identity permutation for any j. Permutation invariance of CNNs The key observation in understanding permutation invariance of CNNs is that instead of neurons, channels define the invariance. To be more concrete, let Conv(x, W) define convolutional operation on input x with weights W ∈ R, where C in, C out are the numbers of input/output channels and w, h are the width and height of the filters. Applying any permutation to the output dimension of the weights and then same permutation to the input channel dimension of the subsequent layer will not change the corresponding CNN's forward pass. Analogous to equation 4 we can write: Note that this formulation permits pooling operations as those act within channels. To apply matched averaging for the nth CNN layer we form inputs to equation 2 as This can be alternatively derived taking the IM2COL perspective. Similar to FCs, we can recursively perform matched averaging on deep CNNs. The immediate consequence of our is the extension of PFNM to CNNs. Empirically (Figure 1, see One-Shot Matching) we found that this extension performs well on MNIST with a simpler CNN architecture such as LeNet (4 layers) and significantly outperforms coordinate-wise weight averaging (1 round FedAvg). However, it breaks down for more complex architecture, e.g. VGG-9 (9 layers), needed to obtain good quality prediction on a more challenging CIFAR-10. Permutation invariance of LSTMs Permutation invariance in the recurrent architectures is associated with the ordering of the hidden states. At a first glance it appears similar to fully connected architecture, however the important difference is associated with the permutation invariance of the hidden-to-hidden weights H ∈ R L×L, where L is the number of hidden states. In particular, permutation of the hidden states affects both rows and columns of H. Consider a basic RNN h t = σ(h t−1 H + x t W), where W are the input-to-hidden weights. To account for the permutation invariance of the hidden states, we notice that dimensions of h t should be permuted in the same way for any t, hence To match RNNs, the basic sub-problem is to align hidden-to-hidden weights of two clients with Euclidean similarity, which requires minimizing Π T H j Π − H j 2 2 over permutations Π. This is a quadratic assignment problem, which is NP-hard. Fortunately, the same permutation appears in an already familiar context of input-to-hidden matching of W Π. Our matched averaging RNN solution is to utilize equation 2 plugging-in input-to-hidden weights {W j} to find {Π j}. Then federated hidden-to-hidden weights are computed as H = 1 J j Π j H h Π T j and input-to-hidden weights are computed as before. LSTMs have multiple cell states, each having its individual hiddento-hidden and input-to-hidden weights. In out matched averaging we stack input-to-hidden weights into SD × L weight matrix (S is the number of cell states; D is input dimension and L is the number of hidden states) when computing the permutation matrices and then average all weights as described previously. LSTMs also often have an embedding layer, which we handle like a fully connected layer. Finally, we process deep LSTMs in the recursive manner similar to deep FCs. Defining the permutation invariance classes of CNNs and LSTMs allows us to extend PFNM to these architectures, however our empirical study in Figure 1 (see OneShot Matching) demonstrates that such extension fails on deep architectures necessary to solve more complex tasks. Our suggest that recursive handling of layers with matched averaging may entail poor overall solution. To alleviate this problem and utilize the strength of matched averaging on "shallow" architectures, we propose the following layer-wise matching scheme. First, data center gathers only the first layer's weights from the clients and performs one-layer matching described previously to obtain the first layers weights of the federated model. Data center then broadcasts these weights to the clients, which proceed to train all consecutive layers on their datasets, keeping the matched federated layers frozen. This procedure is then repeated up to the last layer for which we conduct a weighted averaging based on the class proportions of data points per client. We summarize our Federated Matched Averaging (FedMA) in Algorithm 1. The FedMA approach requires communication rounds equal to the number of layers in a network. In Figure 1 we show that with layer-wise matching FedMA performs well on the deeper VGG-9 CNN as well as LSTMs. In the more challenging heterogeneous setting, FedMA outperforms FedAvg, FedProx trained with same number of communication rounds (4 for LeNet and LSTM and 9 for VGG-9) and other baselines, i.e. client individual CNNs and their ensemble. j p jk W jl,n where p k is fraction of data points with label k on worker j; end for j ∈ {1, . . ., J} do W j,n+1 ← Π T j,n W j,n+1; // permutate the next-layer weights Train {W j,n+1, . . ., W j,L} with W n frozen; end n = n + 1; end FedMA with communication We've shown that in the heterogeneous data scenario FedMA outperforms other federated learning approaches, however it still lags in performance behind the entire data training. Of course the entire data training is not possible under the federated learning constraints, but it serves as performance upper bound we should strive to achieve. To further improve the performance of our method, we propose FedMA with communication, where local clients receive the matched global model at the beginning of a new round and reconstruct their local models with the size equal to the original local models (e.g. size of a VGG-9) based on the matching of the previous round. This procedure allows to keep the size of the global model small in contrast to a naive strategy of utilizing full matched global model as a starting point across clients on every round. We present an empirical study of FedMA with communication and compare it with state-of-the-art methods i.e. FedAvg and FedProx ; analyze the performance under the growing number of clients and visualize the matching behavior of FedMA to study its interpretability. Our experimental studies are conducted over three real world datasets. Summary information about the datasets and associated models can be found in supplement Table 3. Experimental Setup We implemented FedMA and the considered baseline methods in PyTorch . We deploy our empirical study under a simulated federated learning environment where we treat one centralized node in the distributed cluster as the data center and the other nodes as local clients. All nodes in our experiments are deployed on p3.2xlarge instances on Amazon EC2. We assume the data center samples all the clients to join the training process for every communication round for simplicity. For the CIFAR-10 dataset, we use data augmentation (random crops, and flips) and normalize each individual image (details provided in the Supplement). We note that we ignore all batch normalization layers in the VGG architecture and leave it for future work. For CIFAR-10, we considered two data partition strategies to simulate federated learning scenario: (i) homogeneous partition where each local client has approximately equal proportion of each of the classes; (ii) heterogeneous partition for which number of data points and class proportions are unbalanced. We simulated a heterogeneous partition into J clients by sampling p k ∼ Dir J (0.5) and allocating a p k,j proportion of the training instances of class k to local client j. We use the original test set in CIFAR-10 as our global test set and all test accuracy in our experiments are conducted over that test set. For the Shakespeare dataset, since each speaking role in each play is considered a different client according to , it's inherently heterogeneous. We preprocess the Shakespeare dataset by filtering out the clients with datapoints less 10k and get 132 clients in total. We choose 80% of data in training set. We then randomly sample J = 66 out of 132 clients in conducting our experiments. We amalgamate all test sets on clients as our global test set. In this experiment we study performance of FedMA with communication. Our goal is to compare our method to FedAvg and FedProx in terms of the total message size exchanged between data center and clients (in Gigabytes) and the number of communication rounds (recall that completing one FedMA pass requires number of rounds equal to the number of layers in the local models) needed for the global model to achieve good performance on the test data. We also compare to the performance of an ensemble method. We evaluate all methods under the heterogeneous federated learning scenario on CIFAR-10 with J = 16 clients with VGG-9 local models and on Shakespeare dataset with J = 66 clients with 1-layer LSTM network. We fix the total rounds of communication allowed for FedMA, FedAvg, and FedProx i.e. 11 rounds for FedMA and 99/33 rounds for FedAvg and FedProx for the VGG-9/LSTM experiments respectively. We notice that local training epoch is a common parameter shared by the three considered methods, we thus tune the local training epochs (we denote it by E) (comprehensive analysis will be presented in the next experiment) and report the convergence rate under the best E that yields the best final model accuracy over the global test set. We also notice that there is another hyper-parameter in FedProx i.e. the coefficient µ associated with the proxy term, we also tune the parameter using grid search and report the best µ we found i.e. 0.001 for both VGG-9 and LSTM experiments. FedMA outperforms FedAvg and FedProx in all scenarios (Figure 2) with its advantage especially pronounced when we evaluate convergence as a function of the message size in Figures 2(a) and 2(c). Final performance of all trained models is summarized in Tables 1 and 2. Effect of local training epochs As studied in previous work (; ;), the number of local training epochs E can affect the performance of FedAvg and sometimes lead to divergence. We conduct an experimental study on the effect of E over FedAvg, FedProx, and FedMA on VGG-9 trained on CIFAR-10 under heterogeneous setup. The candidate local epochs we considered are E ∈ {10, 20, 50, 70, 100, 150}. For each of the candidate E, we run FedMA for 6 rounds while FedAvg and FedProx for 54 rounds and report the final accuracy that each methods achieves. The is shown in Figure 3. We observed that training longer favors the convergence rate of FedMA, which matches the our assumption that FedMA returns better global model on local models with higher quality. For FedAvg, longer local training leads to deterioration of the final accuracy, which matches the observation in the previous literature (; ;). FedProx prevents the accuracy deterioration to some extent, however, the accuracy of final model still gets reduced. The of this experiment suggests that FedMA is the only method that local clients can use to train their model as long as they want. Handling data bias Real world data often exhibit multimodality within each class, e.g. geodiversity. It has been shown that an observable amerocentric and eurocentric bias is present in the widely used ImageNet dataset . Classifiers trained on such data "learn" these biases and perform poorly on the under-represented domains (modalities) since correlation between the corresponding dominating domain and class can prevent the classifier from learning meaningful relations between features and classes. For example, classifier trained on amerocentric and eurocentric data may learn to associate white color dress with a "bride" class, therefore underperforming on the wedding images taken in countries where wedding traditions are different . The data bias scenario is an important aspect of federated learning, however it received little to no attention in the prior federated learning works. In this study we argue that FedMA can handle this type of problem. If we view each domain, e.g. geographic region, as one client, local models will not be affected by the aggregate data biases and learn meaningful relations between features and classes. FedMA can then be used to learn a good global model without biases. We have already demonstrated strong performance of FedMA on federated learning problems with heterogeneous data across clients and this scenario is very similar. To verify this conjecture we conduct the following experiment. We simulate the skewed domain problem with CIFAR-10 dataset by randomly selecting 5 classes and making 95% training images in those classes to be grayscale. For the remaining 5 we turn only 5% of the corresponding images into grayscale. By doing so, we create 5 grayscale images dominated classes and 5 colored images dominated classes. In the test set, there is half grayscale and half colored images for each class. We anticipate entire data training to pick up the uninformative correlations between greyscale and certain classes, leading to poor test performance without these correlations. In Figure 4 we see that entire data training performs poorly in comparison to the regular (i.e. No Bias) training and testing on CIFAR-10 dataset without any grayscaling. This experiment was motivated by Olga Russakovsky's talk at ICML 2019. Next we compare the federated learning based approaches. We split the images from color dominated classes and grayscale dominated classes into 2 clients. We then conduct FedMA with communication, FedAvg, and FedProx with these 2 clients. FedMA noticeably outperforms the entire data training and other federated learning approach as shown in Figure 4. This suggests that FedMA may be of interest beyond learning under the federated learning constraints, where entire data training is the performance upper bound, but also to eliminate data biases and outperform entire data training. We consider two additional approaches to eliminate data bias without the federated learning constraints. One way to alleviate data bias is to selectively collect more data to debias the dataset. In the context of our experiment, this means getting more colored images for grayscale dominated classes and more grayscale images for color dominated classes. We simulate this scenario by simply doing a full data training where each class in both train and test images has equal amount of grayscale and color images. This procedure, Color Balanced, performs well, but selective collection of new data in practice may be expensive or even not possible. Instead of collecting new data, one may consider oversampling from the available data to debias. In Oversampling, we sample the underrepresented domain (via sampling with replacement) to make the proportion of color and grayscale images to be equal for each class (oversampled images are also passed through the data augmentation pipeline, e.g. random flipping and cropping, to further enforce the data diversity). Such procedure may be prone to overfitting the oversampled images and we see that this approach only provides marginal improvement of the model accuracy compared to centralized training over the skewed dataset and performs noticeably worse than FedMA. Data efficiency It is known that deep learning models perform better when more training data is available. However, under the federated learning constraints, data efficiency has not been studied to the best of our knowledge. The challenge here is that when new clients join the federated system, they each bring their own version of the data distribution, which, if not handled properly, may deteriorate the performance despite the growing data size across the clients. To simulate this scenario we first partition the entire training CIFAR-10 dataset into 5 homogeneous pieces. We then partition each homogeneous data piece further into 5 sub-pieces heterogeneously. Using this strategy, we partition the CIFAR-10 training set into 25 heterogeneous small sub-datasets containing approximately 2k points each. We conduct a 5-step experimental study: starting from a randomly selected homogeneous piece consisting of 5 associated heterogeneous sub-pieces, we simulate a 5-client federated learning heterogeneous problem. For each consecutive step, we add one of the remaining homogeneous data pieces consisting of 5 new clients with heterogeneous sub-datasets. Results are presented in Figure 5. Performance of FedMA (with a single pass) improves when new clients are added to the federated learning system, while FedAvg with 9 communication rounds deteriorates. Interpretability One of the strengths of FedMA is that it utilizes communication rounds more efficiently than FedAvg. Instead of directly averaging weights element-wise, FedMA identifies matching groups of convolutional filters and then averages them into the global convolutional filters. It's natural to ask "How does the matched filters look like?". In Figure 6 we visualize the representations generated by a pair of matched local filters, aggregated global filter, and the filter returned by the FedAvg method over the same input image. Matched filters and the global filter found with FedMA are extracting the same feature of the input image, i.e. filter 0 of client 1 and filter 23 of client 2 are extracting the position of the legs of the horse, and the corresponding matched global filter 0 does the same. For the FedAvg, global filter 0 is the average of filter 0 of client 1 and filter 0 of client 2, which clearly tampers the leg extraction functionality of filter 0 of client 1. In this paper, we presented FedMA, a new layer-wise federated learning algorithm designed for modern CNNs and LSTMs architectures utilizing probabilistic matching and model size adaptation. We demonstrate the convergence rate and communication efficiency of FedMA empirically. In the future, we would like to extend FedMA towards finding the optimal averaging strategy. Making FedMa support more building blocks e.g. residual structures in CNNs and batch normalization layers is also of interest. Table 4: Detailed information of the VGG-9 architecture used in our experiments, all non-linear activation function in this architecture is ReLU; the shapes for convolution layers follows (Cin, Cout, c, c) In preprocessing the images in CIFAR-10 dataset, we follow the standard data augmentation and normalization process. For data augmentation, random cropping and horizontal random flipping are used. Each color channels are normalized with mean and standard deviation by µ r = 0.491372549, µ g = 0.482352941, µ b = 0.446666667, σ r = 0.247058824, σ g = 0.243529412, σ b = 0.261568627. Each channel pixel is normalized by subtracting the mean value in this color channel and then divided by the standard deviation of this color channel. Here we report the shapes of final global VGG and LSTM models returned by FRB with communication.

Represent the short summary for finding the corresponding scientific article (long)

Communication efficient federated learning with layer-wise matching

Represent the TLDR for finding the corresponding scientific article

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.