Question: Represent the scientific article to find a short TLDR

Question: Model-free reinforcement learning (RL) methods are succeeding in a growing number of tasks, aided by recent advances in deep learning. However, they tend to suffer from high sample complexity, which hinders their use in real-world domains. Alternatively, model-based reinforcement learning promises to reduce sample complexity, but tends to require careful tuning and to date have succeeded mainly in restrictive domains where simple models are sufficient for learning. In this paper, we analyze the behavior of vanilla model-based reinforcement learning methods when deep neural networks are used to learn both the model and the policy, and show that the learned policy tends to exploit regions where insufficient data is available for the model to be learned, causing instability in training. To overcome this issue, we propose to use an ensemble of models to maintain the model uncertainty and regularize the learning process. We further show that the use of likelihood ratio derivatives yields much more stable learning than backpropagation through time. Altogether, our approach Model-Ensemble Trust-Region Policy Optimization (ME-TRPO) significantly reduces the sample complexity compared to model-free deep RL methods on challenging continuous control benchmark tasks. Deep reinforcement learning has achieved many impressive in recent years, including learning to play Atari games from raw-pixel inputs BID0, mastering the game of Go BID1, as well as learning advanced locomotion and manipulation skills from raw sensory inputs BID3 BID4. Many of these were achieved using model-free reinforcement learning algorithms, which do not attempt to build a model of the environment. These algorithms are generally applicable, require relatively little tuning, and can easily incorporate powerful function approximators such as deep neural networks. However, they tend to suffer from high sample complexity, especially when such powerful function approximators are used, and hence their applications have been mostly limited to simulated environments. In comparison, model-based reinforcement learning algorithms utilize a learned model of the environment to assist learning. These methods can potentially be much more sample efficient than model-free algorithms, and hence can be applied to real-world tasks where low sample complexity is crucial BID7 BID3 BID8. However, so far such methods have required very restrictive forms of the learned models, as well as careful tuning for them to be applicable. Although it is a straightforward idea to extend model-based algorithms to deep neural network models, so far there has been comparatively fewer successful applications. The standard approach for model-based reinforcement learning alternates between model learning and policy optimization. In the model learning stage, samples are collected from interaction with the environment, and supervised learning is used to fit a dynamics model to the observations. In the policy optimization stage, the learned model is used to search for an improved policy. The underlying assumption in this approach, henceforth termed vanilla model-based RL, is that with enough data, the learned model will be accurate enough, such that a policy optimized on it will also perform well in the real environment. Although vanilla model-based RL can work well on low-dimensional tasks with relatively simple dynamics, we find that on more challenging continuous control tasks, performance was highly unstable. The reason is that the policy optimization tends to exploit regions where insufficient data is available to train the model, leading to catastrophic failures. Previous work has pointed out this issue as model bias, i.e. BID7 BID9 BID10. While this issue can be regarded as a form of overfitting, we emphasize that standard countermeasures from the supervised learning literature, such as regularization or cross validation, are not sufficient here -supervised learning can guarantee generalization to states from the same distribution as the data, but the policy optimization stage steers the optimization exactly towards areas where data is scarce and the model is inaccurate. This problem is severely aggravated when expressive models such as deep neural networks are employed. To resolve this issue, we propose to use an ensemble of deep neural networks to maintain model uncertainty given the data collected from the environment. During model learning, we differentiate the neural networks by varying their weight initialization and training input sequences. Then, during policy learning, we regularize the policy updates by combining the gradients from the imagined stochastic roll-outs. Each imagined step is uniformly sampled from the ensemble predictions. Using this technique, the policy learns to become robust against various possible scenarios it may encounter in the real environment. To avoid overfitting to this regularized objective, we use the model ensemble for early stopping policy training. Standard model-based techniques require differentiating through the model over many time steps, a procedure known as backpropagation through time (BPTT). It is well-known in the literature that BPTT can lead to exploding and vanishing gradients BID11. Even when gradient clipping is applied, BPTT can still get stuck in bad local optima. We propose to use likelihood ratio methods instead of BPTT to estimate the gradient, which only make use of the model as a simulator rather than for direct gradient computation. In particular, we use Trust Region Policy Optimization (TRPO) BID4, which imposes a trust region constraint on the policy to further stabilize learning. In this work, we propose Model-Ensemble Trust-Region Policy Optimization (ME-TRPO), a modelbased algorithm that achieves the same level of performance as state-of-the-art model-free algorithms with 100× reduction in sample complexity. We show that the model ensemble technique is an effective approach to overcome the challenge of model bias in model-based reinforcement learning. We demonstrate that replacing BPTT by TRPO yields significantly more stable learning and much better final performance. Finally, we provide an empirical analysis of vanilla model-based RL using neural networks as function approximators, and identify its flaws when applied to challenging continuous control tasks. There has been a large body of work on model-based reinforcement learning. They differ by the choice of model parameterization, which is associated with different ways of utilizing the model for policy learning. Interestingly, the most impressive robotic learning applications so far were achieved using the simplest possible model parameterization, namely linear models BID12 BID13 BID14 BID15 BID3 BID16, where the model either operates directly over the raw state, or over a feature representation of the state. Such models are very data efficient, and allows for very efficient policy optimization through techniques from optimal control. However, they only have limited expressiveness, and do not scale well to complicated nonlinear dynamics or high-dimensional state spaces, unless a separate feature learning phase is used BID15 ).An alternative is to use nonparametric models such as Gaussian Processes (GPs) BID17 BID18 BID7. Such models can effectively maintain uncertainty over the predictions, and have infinite representation power as long as enough data is available. However, they suffer from the curse of dimensionality, and so far their applications have been limited to relatively low-dimensional settings. The computational expense of incorporating the uncertainty estimates from GPs into the policy update also imposes an additional challenge. Deep neural networks have shown great success in scaling up model-free reinforcement learning algorithms to challenging scenarios BID0 BID1 BID4. However, there has been only limited success in applying them to model-based RL. Although many previous studies have shown promising on relatively simple domains BID19 BID20 BID21 BID22, so far their applications on more challenging domains have either required a combination with model-free techniques BID23 BID25, or domain-specific policy learning or planning algorithms BID26 BID27 BID28 BID29 BID30 BID31. In this work, we show that our purely model-based approach improves the sample complexity compared to methods that combine model-based and model-free elements. Two recent studies have shown promising signs towards a more generally applicable model-based RL algorithm. BID32 utilize Bayesian neural networks (BNNs) to learn a distribution over dynamics models, and train a policy using gradient-based optimization over a collection of models sampled from this distribution. BID33 learn a latent variable dynamic model over temporally extended segments of the trajectory, and train a policy using gradient-based optimization over the latent space. Both of these approaches have been shown to work on a fixed dataset of samples which are collected before the algorithm starts operating. Hence, their evaluations have been limited to domains where random exploration is sufficient to collect data for model learning. In comparison, our approach utilizes an iterative process of alternatively performing model learning and policy learning, and hence can be applied to more challenging domains. Additionally, our proposed improvements are orthogonal to both approaches, and can be potentially combined to yield even better . This paper assumes a discrete-time finite-horizon Markov decision process (MDP), defined by (S, A, f, r, ρ 0, T), in which S ⊆ R n is the state space, A ⊆ R m the action space, f: S × A → S a deterministic transition function, r: S × A → R a bounded reward function, ρ 0: S → R + an initial state distribution, and T the horizon. We denote a stochastic policy π θ (a|s) as the probability of taking action a at state s. Let η(θ) denote its expected return: DISPLAYFORM0 where τ = (s 0, a 0, . . ., a T −1, s T) denotes the whole trajectory, s 0 ∼ ρ 0 , a t ∼ π θ (.|s t), and s t+1 = f (s t, a t) for all t. We assume that the reward function is known but the transition function is unknown. Our goal is to find an optimal policy that maximizes the expected return η(θ). In the most successful methods of model-free reinforcement learning, we iteratively collect data, estimate the gradient of the policy, improve the policy, and then discard the data. Conversely, modelbased reinforcement learning makes more extensive use of the data; it uses all the data collected to train a model of the dynamics of the environment. The trained model can be used as a simulator in which the policy can be trained, and also provides gradient information BID34 BID7 BID32 BID35. In the following section, we describe the vanilla model-based reinforcement learning algorithm (see Algorithm 1). We assume that the model and the policy are represented by neural networks, but the methodology is valid for other types of function approximators. The transition dynamics is modeled with a feed-forward neural network, using the standard practice to train the neural network to predict the change in state (rather than the next state) given a state and an action as inputs. This relieves the neural network from memorizing the input state, especially when the change is small BID7 BID36 BID25.We denote the function approximator for the next state, which is the sum of the input state and the output of the neural network, asf φ.The objective of model learning is to find a parameter φ that minimizes the L 2 one-step prediction loss 2: DISPLAYFORM0 where D is the training dataset that stores the transitions the agent has experienced. We use the Adam optimizer BID38 to solve this supervised learning problem. Standard techniques are followed to avoid overfitting and facilitate the learning such as separating a validation dataset to early stop the training, and normalizing the inputs and outputs of the neural network Given an MDP, M, the goal of reinforcement learning is to maximize the expected sum of rewards. During training, model-based methods maintain an approximate MDP,M, where the transition function is given by a parameterized modelf φ learned from data. The policy is then updated with respect to the approximate MDP. Hence, the objective we maximize iŝ DISPLAYFORM1 DISPLAYFORM2, and s t+1 =f φ (s t, a t).We represent the stochastic policy 4 as a conditional multivariate normal distribution with a parametrized mean µ θ: S → A and a parametrized standard deviation σ θ: S → R m. Using the re-parametrization trick, we can write down an action sampled from π θ at state s as µ θ (s) + σ θ (s)T ζ, where ζ ∼ N (0, I m). Given a trajectoryτ sampled using the policy π θ, we can recover the noise vectors {ζ 0, ..., ζ T}. Thus, the gradient of the objectiveη(θ; φ) can simply be estimated by Monte-Carlo methods: DISPLAYFORM3 This method of gradient computation is called backpropagation through time (BPTT), which can be easily performed using an automatic differentiation library. We apply gradient clipping BID39 to deal with exploding gradients, and we use the Adam optimizer BID38 for more stable learning. We perform the updates until the policy no longer improves its estimated performanceη over a period of time (controlled by a hyperparameter), and then we repeat the process in the outer loop by using the policy to collect more data with respect to the real model 5. The whole procedure terminates when the desired performance according to the real model is accomplished. Using the vanilla approach described in Section 4, we find that the learned policy often exploits regions where scarce training data is available for the dynamics model. Since we are improving Collect samples from the real environment f using π θ and add them to D. Train the modelf φ using D.6: DISPLAYFORM0 Collect fictitious samples fromf φ using π θ. Update the policy using BPTT on the fictitious samples. Estimate the performanceη(θ; φ). until the performance stop improving. 11: until the policy performs well in real environment f. the policy with respect to the approximate MDP instead of the real one, the predictions then can be erroneous to the policy's advantage. This overfitting issue can be partly alleviated by early stopping using validation initial states in a similar manner to supervised learning. However, we found this insufficient, since the performance is still evaluated using the same learned model, which tends to make consistent mistakes. Furthermore, although gradient clipping can usually resolve exploding gradients, BPTT still suffers from vanishing gradients, which cause the policy to get stuck in bad local optima BID11 BID39. These problems are especially aggravated when optimizing over long horizons, which is very common in reinforcement learning problems. We now present our method, Model-Ensemble Trust-Region Policy Optimization (ME-TRPO). The pseudocode is shown in Algorithm 2. ME-TRPO combines three modifications to the vanilla approach. First, we fit a set of dynamics models {f φ1, . . ., f φ K} (termed a model ensemble) using the same real world data. These models are trained via standard supervised learning, as described in Section 4.1, and they only differ by the initial weights and the order in which mini-batches are sampled. Second, we use Trust Region Policy Optimization (TRPO) to optimize the policy over the model ensemble. Third, we use the model ensemble to monitor the policy's performance on validation data, and stops the current iteration when the policy stops improving. The second and third modifications are described in detail below. Policy Optimization. To overcome the issues with BPTT, we use likelihood-ratio methods from the model-free RL literature. We evaluated using Vanilla Policy Gradient (VPG) BID40, Proximal Policy Optimization (PPO), and Trust Region Policy Optimization (TRPO) BID4. The best were achieved by TRPO. In order to estimate the gradient, we use the learned models to simulate trajectories as follows: in every step, we randomly choose a model to predict the next state given the current state and action. This avoids the policy from overfitting to any single model during an episode, leading to more stable learning. Policy Validation. We monitor the policy's performance using the K learned models. Specifically, we compute the ratio of models in which the policy improves: DISPLAYFORM0 The current iteration continues as long as this ratio exceeds a certain threshold. In practice, we validate the policy after every 5 gradient updates and we use 70% as the threshold. If the ratio falls below the threshold, a small number of updates is tolerated in case the performance improves, the current iteration is terminated. Then, we repeat the overall process of using the policy to collect more real-world data, optimize the model ensemble, and using the model ensemble to improve the policy. This process continues until the desired performance in the real environment is reached. The model ensemble serves as effective regularization for policy learning: by using the model ensemble for policy optimization and validation, the policy is forced to perform well over a vast number of possible alternative futures. Even though any of the individual models can still incur model bias, our experiments below suggest that combining these models yields stable and effective policy improvement. Collect samples from the real system f using π θ and add them to D. Train all models using D. repeatOptimize π θ using all models. Collect fictitious samples from {f φi} K i=1 using π θ. Update the policy using TRPO on the fictitious samples. Estimate the performancesη(θ; φ i) for i = 1,..., K. until the performances stop improving. 11: until the policy performs well in real environment f. We design the experiments to answer the following questions:1. How does our approach compare against state-of-the-art methods in terms of sample complexity and final performance? 2. What are the failure scenarios of the vanilla algorithm? 3. How does our method overcome these failures?We also provide in the Appendix D an ablation study to characterize the effect of each component of our algorithm. To answer these questions, we evaluate our method and various baselines over six standard continuous control benchmark tasks BID42 BID43 in Mujoco BID44: Swimmer, Snake, Hopper, Ant, Half Cheetah, and Humanoid, shown in FIG1. The details of the tasks can be found in Appendix A.2. We compare our method with the following state-of-the-art reinforcement learning algorithms in terms of sample complexity and performance: Trust Region Policy Optimization (TRPO) BID4, Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), and Stochastic Value Gradient (SVG).The are shown in FIG2. Prior model-based methods appear to achieve worse performance compared with model-free methods. In addition, we find that model-based methods tend to be difficult to train over long horizons. In particular, SVG, not presented in the plots, is very unstable in our experiments. While SVG(∞) is more stable, it fails to achieve the same level of performance as model-free methods. In contrast, our proposed method reaches the same level of performance as model-free approaches with ≈ 100× less data. To the best of our knowledge, it is the first purely model-based approach that can optimize policies over high-dimensional motor-control tasks such as Humanoid. For experiment details please refer to Appendix A. In this section we explain and quantify the failure cases of vanilla model-based reinforcement learning, and how our approach overcomes such failures. We analyze the effect of each of our proposed modifications by studying the learning behavior of replacing BPTT with TRPO in vanilla model-based RL using just a single model, and then the effect of using an ensemble of models. As discussed above, BPTT suffers from exploding and vanishing gradients, especially when optimizing over long horizons. Furthermore, one of the principal drawbacks of BPTT is the assumption that the model derivatives should match that of the real dynamics, even though the model has not been explicitly trained to provide accurate gradient information. In FIG3 we demonstrate the effect of using policy gradient methods that make use of a score function estimator, such as VPG and TRPO, while using a single learned model. The suggest that in comparison with BPTT, policy gradient methods are more stable and lead to much better final performance. By using such model-free algorithms, we require less information from the learned model, which only acts as a simulator. Gradient information through the dynamics model is not needed anymore to optimize the policy. However, while replacing BPTT by TRPO helps optimization, the learned policy can still suffer from model bias. The learning procedure tends to steer the policy towards regions where it has rarely visited, so that the model makes erroneous predictions to its advantage. The estimated performances of the policy often end up with high rewards according to the learned model, and low rewards according to the real one (see Appendix B for further discussion). In Figure 4, we analyze the effect of using various numbers of ensemble models for sampling trajectories and validating the policy's performance. The indicate that as more models are used in the model ensemble, the learning is better regularized and the performance continually improves. The improvement is even more noticeable in more challenging environments like HalfCheetah and Ant, which require more complex dynamics models to be learned, leaving more room for the policy to exploit when model ensemble is not used. Figure 4: Comparison among different number of models that the policy is trained on. TRPO is used for the policy optimization. We illustrate the improvement when using 5, 10 and 20 models over a single model (Best viewed in color). In this work, we present a simple and robust model-based reinforcement learning algorithm that is able to learn neural network policies across different challenging domains. We show that our approach significantly reduces the sample complexity compared to state-of-the-art methods while reaching the same level of performance. In comparison, our analyses suggests that vanilla model-based RL tends to suffer from model bias and numerical instability, and fails to learn a good policy. We further evaluate the effect of each key component of our algorithm, showing that both using TRPO and model ensemble are essential for successful applications of deep model-based RL. We also confirm the of previous work BID7 BID32 BID22 that using model uncertainty is a principled way to reduce model bias. One question that merits future investigation is how to use the model ensemble to encourage the policy to explore the state space where the different models disagree, so that more data can be collected to resolve their disagreement. Another enticing direction for future work would be the application of ME-TRPO to real-world robotics systems. Our algorithm can be broken down into three parts: data collection, model learning, and policy learning. We describe the numerical details for each part below. A.1.1 DATA COLLECTION In each outer iteration, we use the stochastic policy to collect 3000 timesteps of real world data for every environment, except Humanoid in which we collect 6000 timesteps. At the beginning of every roll-out we sample the policy standard deviation randomly from U[0.0, 3.0], and we keep the value fixed throughout the episode. Furthermore, we perturb the policy's parameters by adding white Gaussian noise with standard deviation proportional to the absolute difference between the current parameters and the previous one BID45 BID46. Finally, we split the collected data using a 2-to-1 ratio for training and validation datasets. We represent the dynamics model with a 2-hidden-layer feed-forward neural network with hidden sizes 1024-1024 and ReLU nonlinearities. We train the model with the Adam optimizer with learning rate 0.001 using a batch size of 1000. The model is trained until the validation loss has not decreased for 25 passes over the entire training dataset (we validate the training every 5 passes). We represent the policy with a 2-hidden-layer feed-forward neural network with hidden sizes 32-32 and tanh nonlinearities for all the environments, except Humanoid, in which we use the hidden sizes 100-50-25. The policy is trained with TRPO on the learned models using initial standard deviation 1.0, step size δ KL 0.01, and batch size 50000. If the policy fails the validation for 25 updates (we do the validation every 5 updates), we stop the learning and repeat the overall process. The environments we use are adopted from rllab BID42 Note that in Hopper we relax the early stopping criterion to a soft constraint in reward function, whereas in Ant we early stop when the center of mass long z-axis is outside [0.2, 1.0] and have a survival reward when alive. The state in each environment is composed of the joint angles, the joint velocities, and the cartesian position of the center of mass of a part of the simulated robot. We are not using the contact information, which make the environments effectively POMDPs in Half Cheetah, Ant, Hopper and Humanoid. We also eliminate the redundancies in the state space in order to avoid infeasible states in the prediction. In Section 6.2 we compare our method against TRPO, PPO, DDPG, and SVG. For every environment we represent the policy with a feed-forward neural network of the same size, horizon, and discount factor as the ones specified in the Appendix A.1.3. In the following we provide the hyper-parameters details:Trust Region Policy Optimization. We used the implementation of BID42 with a batch size of 50000, and we train the policies for 1000 iterations. The step size δ KL that we used in all the experiments was of 0.05.Proximal Policy Optimization. We referred to the implementation of BID43. The policies were trained for 10 7 steps using the default hyper-parameters across all tasks. Deep Deterministic Policy Gradient. We also use the implementation of BID43 using a number epochs of 2000, the rest of the hyper-parameters used were the default ones. Stochastic Value Gradient. We parametrized the dynamics model as a feedforward neural network of two hidden layers of 512 units each and ReLU non-linearities. The model was trained after every episode with the data available in the replay buffer, using the Adam optimizer with a learning rate of 10 −4, and batch size of 128. We additionally clipped the gradient we the norm was larger than 10. We show that replacing the ensemble with just one model leads to the policy overoptimization. In each outer iteration, we see that at the end of the policy optimization step the estimated performance increases while the real performance is in fact decreasing (see FIG4). C REAL-TIME COMPLEXITY We provide wall clock time for the ME-TRPO from figure 2 in the table below: Finally, we provide a study of the different ways for validating the policy. We compare the following techniques: using the real performance (i.e., using an oracle) (real), using the average return in the trpo roll-outs (trpo mean), stopping the policy after 50 gradient updates (no early 50), or after 5 gradient updates (no early 5), using one model to predict the performances (one model), and using an ensemble of models (ensemble). The experiments are designed to use the same number of models and hyper-parameters for the other components of the algorithm. In FIG5 we can see the effectiveness of each approach. It is noteworthy that having an oracle of the real performance is not the best approach. Such validation is over-cautious, and does not give room for exploration ing in a poor trained dynamics model. Stopping the gradient after a fixed number of updates in good performance if the right number of updates is set. This burdens the hyper-parameter search with one more hyper-parameter. On the other hand, using the ensemble of models has good performance across environments without adding extra hyper-parameters.

Represent the TLDR for finding the corresponding scientific article

We propose Dual Actor-Critic algorithm, which is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation. The algorithm achieves the state-of-the-art performances across several benchmarks.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: The high computational and parameter complexity of neural networks makes their training very slow and difficult to deploy on energy and storage-constrained comput- ing systems. Many network complexity reduction techniques have been proposed including fixed-point implementation. However, a systematic approach for design- ing full fixed-point training and inference of deep neural networks remains elusive. We describe a precision assignment methodology for neural network training in which all network parameters, i.e., activations and weights in the feedforward path, gradients and weight accumulators in the feedback path, are assigned close to minimal precision. The precision assignment is derived analytically and enables tracking the convergence behavior of the full precision training, known to converge a priori. Thus, our work leads to a systematic methodology of determining suit- able precision for fixed-point training. The near optimality (minimality) of the ing precision assignment is validated empirically for four networks on the CIFAR-10, CIFAR-100, and SVHN datasets. The complexity reduction arising from our approach is compared with other fixed-point neural network designs. Though deep neural networks (DNNs) have established themselves as powerful predictive models achieving human-level accuracy on many machine learning tasks BID12, their excellent performance has been achieved at the expense of a very high computational and parameter complexity. For instance, AlexNet BID17 requires over 800 × 10 6 multiply-accumulates (MACs) per image and has 60 million parameters, while Deepface requires over 500 × 10 6 MACs/image and involves more than 120 million parameters. DNNs' enormous computational and parameter complexity leads to high energy consumption BID4, makes their training via the stochastic gradient descent (SGD) algorithm very slow often requiring hours and days BID9, and inhibits their deployment on energy and resource-constrained platforms such as mobile devices and autonomous agents. A fundamental problem contributing to the high computational and parameter complexity of DNNs is their realization using 32-b floating-point (FL) arithmetic in GPUs and CPUs. Reduced-precision representations such as quantized FL (QFL) and fixed-point (FX) have been employed in various combinations to both training and inference. Many employ FX during inference but train in FL, e.g., fully binarized neural networks BID13 use 1-b FX in the forward inference path but the network is trained in 32-b FL. Similarly, BID10 employs 16-b FX for all tensors except for the internal accumulators which use 32-b FL, and 3-level QFL gradients were employed (; BID0 to accelerate training in a distributed setting. Note that while QFL reduces storage and communication costs, it does not reduce the computational complexity as the arithmetic remains in 32-b FL.Thus, none of the previous works address the fundamental problem of realizing true fixed-point DNN training, i.e., an SGD algorithm in which all parameters/variables and all computations are implemented in FX with minimum precision required to guarantee the network's inference/prediction accuracy and training convergence. The reasons for this gap are numerous including: 1) quantization Step 1: Forward PropagationStep 2: Back PropagationStep 3: Update errors propagate to the network output thereby directly affecting its accuracy ; 2) precision requirements of different variables in a network are interdependent and involve hard-toquantify trade-offs ; 3) proper quantization requires the knowledge of the dynamic range which may not be available ; and 4) quantization errors may accumulate during training and can lead to stability issues BID10.Our work makes a major advance in closing this gap by proposing a systematic methodology to obtain close-to-minimum per-layer precision requirements of an FX network that guarantees statistical similarity with full precision training. In particular, we jointly address the challenges of quantization noise, inter-layer and intra-layer precision trade-offs, dynamic range, and stability. As in , we do assume that a fully-trained baseline FL network exists and one can observe its learning behavior. While, in principle, such assumption requires extra FL computation prior to FX training, it is to be noted that much of training is done in FL anyway. For instance, FL training is used in order to establish benchmarking baselines such as AlexNet BID17, VGG-Net , and ResNet BID12, to name a few. Even if that is not the case, in practice, this assumption can be accounted for via a warm-up FL training on a small held-out portion of the dataset BID6.Applying our methodology to three benchmarks reveals several lessons. First and foremost, our work shows that it is possible to FX quantize all variables including back-propagated gradients even though their dynamic range is unknown BID15. Second, we find that the per-layer weight precision requirements decrease from the input to the output while those of the activation gradients and weight accumulators increase. Furthermore, the precision requirements for residual networks are found to be uniform across layers. Finally, hyper-precision reduction techniques such as weight and activation binarization BID13 or gradient ternarization are not as efficient as our methodology since these do not address the fundamental problem of realizing true fixed-point DNN training. We demonstrate FX training on three deep learning benchmarks (CIFAR-10, CIFAR-100, SVHN) achieving high fidelity to our FL baseline in that we observe no loss of accuracy higher then 0.56% in all of our experiments. Our precision assignment is further shown to be within 1-b per-tensor of the minimum. We show that our precision assignment methodology reduces representational, computational, and communication costs of training by up to 6×, 8×, and 4×, respectively, compared to the FL baseline and related works. We consider a L-layer DNN deployed on a M -class classification task using the setup in FIG1. We denote the precision configuration as the DISPLAYFORM0 whose l th row consists of the precision (in bits) of weight DISPLAYFORM1 ), and internal weight accumulator W DISPLAYFORM2 ) tensors at layer l. This DNN quantization setup is summarized in Appendix A. We present definitions/constraints related to fixed-point arithmetic based on the design of fixed-point adaptive filters and signal processing systems :• A signed fixed-point scalar a with precision B A and binary representation DISPLAYFORM0 2 −i a i, where r A is the predetermined dynamic range (PDR) of a. The PDR is constrained to be a constant power of 2 to minimize hardware overhead.• An unsigned fixed-point scalar a with precision B A and binary representation DISPLAYFORM1 • The precision B A is determined as: B A = log 2 r A ∆ A + 1, where ∆ A is the quantization step size which is the value of the least significant bit (LSB).• An additive model for quantization is assumed: a =ã + q a, where a is the fixed-point number obtained by quantizing the floating-point scalarã, q a is a random variable uniformly distributed on the interval − 12. The notion of quantization noise is most useful when there is limited knowledge of the distribution ofã.• The relative quantization bias η A is the offset: DISPLAYFORM2, where the first unbiased quantization level µ A = E ã ã ∈ I 1 and DISPLAYFORM3. The notion of quantization bias is useful when there is some knowledge of the distribution ofã.• The reflected quantization noise variance from a tensor T to a scalar α = f (T), for an arbitrary function f , is: DISPLAYFORM4, where ∆ T is the quantization step of T and E T →α is the quantization noise gain from T to α.• The clipping rate β T of a tensor T is the probability: β T = Pr ({|t| ≥ r T : t ∈ T}), where r T is the PDR of T. We use a set of metrics inspired by those introduced by which have also been used by Wu et al. (2018a). These metrics are algorithmic in nature which makes them easily reproducible.• Representational Cost for weights (C W) and activations (C A): DISPLAYFORM0, which equals the total number of bits needed to represent the weights, weight gradients, and internal weight accumulators (C W), and those for activations and activation gradients (C A). DISPLAYFORM1, where D l is the dimensionality of the dot product needed to compute one output activation at layer l. This cost is a measure of the number of 1-b full adders (FAs) utilized for all multiplications in one back-prop iteration. DISPLAYFORM2, which represents cost of communicating weight gradients in a distributed setting (; BID0 . We aim to obtain a minimal or close-to-minimal precision configuration C o of a FX network such that the mismatch probability p m = Pr{Ŷ f l =Ŷ f x} between its predicted label (Ŷ f x) and that of an associated FL network (Ŷ f l) is bounded, and the convergence behavior of the two networks is similar. Hence, we require that: all quantization noise sources in the forward path contribute identically to the mismatch budget p m , the gradients be properly clipped in order to limit the dynamic range , the accumulation of quantization noise bias in the weight updates be limited BID10, the quantization noise in activation gradients be limited as these are back-propagated to calculate the weight gradients, and the precision of weight accumulators should be set so as to avoid premature stoppage of convergence BID7. The above insights can be formally described via the following five quantization criteria. Criterion 1. Equalizing Feedforward Quantization Noise (EFQN) Criterion. The reflected quantization noise variances onto the mismatch probability p m from all feedforward weights DISPLAYFORM0 Criterion 2. Gradient Clipping (GC) Criterion. The clipping rates of weight ({β DISPLAYFORM1) gradients should be less than a maximum value β 0: DISPLAYFORM2 Criterion 3. Relative Quantization Bias (RQB) Criterion. The relative quantization bias of weight gradients ({η DISPLAYFORM3 DISPLAYFORM4 where Σ l is the total sum of element-wise variances of G (W) l. Criterion 5. Accumulator Stopping (AS) Criterion. The quantization noise of the internal accumulator should be zero, equivalently: DISPLAYFORM5 is the reflected quantization noise variance from W DISPLAYFORM6, its total sum of element-wise variances. Further explanations and motivations behind the above criteria are presented in Appendix B. The following claim ensures the satisfiability of the above criteria. This leads to closed form expressions for the precision requirements we are seeking and completes our methodology. The validity of the claim is proved in Appendix C. Claim 1. Satisfiability of Quantization Criteria. The five quantization criteria (EFQN, GC, RQB, BQN, AS) are satisfied if:• The precisions B W l and B A l are set as follows: DISPLAYFORM7 for l = 1... L, where rnd denotes the rounding operation, E W l →pm and E A l →pm are the weight and activation quantization noise gains at layer l, respectively, B (min) is a reference minimum precision, and DISPLAYFORM8 Published as a conference paper at ICLR 2019• The weight and activation gradients PDRs are lower bounded as follows: DISPLAYFORM9 where σ DISPLAYFORM10 are the largest recorded estimates of the weight and activation gradients DISPLAYFORM11, respectively.• The weight and activation gradients quantization step sizes are upper bounded as follows: DISPLAYFORM12 where σ DISPLAYFORM13 is the largest singular value of the square-Jacobian (Jacobian matrix with squared entries) of G DISPLAYFORM14 l+1.• The accumulator PDR and step size satisfy: DISPLAYFORM15 where γ (min) is the smallest value of the learning rate used during training. Practical considerations: Note that one of the 2L feedforward precisions will equal B (min). The formulas to compute the quantization noise gains are given in Appendix C and require only one forward-backward pass on an estimation set. We would like the EFQN criterion to hold upon convergence; hence, FORMULA19 is computed using the converged model from the FL baseline. For backward signals, setting the values of PDR and LSB is sufficient to determine the precision using the identity B A = log 2 r A ∆ A + 1, as explained in Section 2.1. As per Claim 1, estimates of the second order statistics, e.g., DISPLAYFORM16, of the gradient tensors, are required. These are obtained via tensor spatial averaging, so that one estimate per tensor is required, and updated in a moving window fashion, as is done for normalization parameters in BatchNorm BID14. Furthermore, it might seem that computing the Jacobian in is a difficult task; however, the values of its elements are already computed by the back-prop algorithm, requiring no additional computations (see Appendix C). Thus, the Jacobians (at different layers) are also estimated during training. Due to the typical very large size of modern neural networks, we average the Jacobians spatially, i.e., the activations are aggregated across channels and mini-batches while weights are aggregated across filters. This is again inspired by the work on Batch Normalization BID14 and makes the probed Jacobians much smaller. We conduct numerical simulations in order to illustrate the validity of the predicted precision configuration C o and investigate its minimality and benefits. We employ three deep learning benchmarking datasets: CIFAR-10, CIFAR-100 BID16 SVHN . All experiments were done using a Pascal P100 NVIDIA GPU. We train the following networks:• CIFAR-10 ConvNet: a 9-layer convolutional neural network trained on the CIFAR-10 dataset described as 2 DISPLAYFORM0 where C3 denotes 3 × 3 convolutions, M P 2 denotes 2 × 2 max pooling operation, and F C denotes fully connected layers.• SVHN ConvNet: the same network as the CIFAR-10 ConvNet, but trained on the SVHN dataset.• CIFAR-10 ResNet: a wide deep residual network with ResNet-20 architecture but having 8 times as many channels per layer compared to BID12.• CIFAR-100 ResNet: same network as CIFAR-10 ResNet save for the last layer to match the number of classes in CIFAR-100.A step by step description of the application of our method to the above four networks is provided in Appendix E. We hope the inclusion of these steps would: clarify any ambiguity the reader may have from the previous section and facilitate the reproduction of our . The precision configuration C o, with target p m ≤ 1%, β 0 ≤ 5%, and η 0 ≤ 1%, via our proposed method is depicted in FIG4 for each of the four networks considered. We observe that C o is dependent on the network type. Indeed, the precisions of the two ConvNets follow similar trends as do those the two ResNets. Furthermore, the following observations are made for the ConvNets:• weight precision B W l decreases as depth increases. This is consistent with the observation that weight perturbations in the earlier layers are the most destructive .• the precisions of activation gradients (B G (A) l ) and internal weight accumulators (B W (acc) l ) increases as depth increases which we interpret as follows: the back-propagation of gradients is the dual of the forward-propagation of activations, and accumulators store the most information as their precision is the highest.• the precisions of the weight gradients (B G (W) l ) and activations (B A l) are relatively constant across layers. Interestingly, for ResNets, the precision is mostly uniform across the layers. Furthermore, the gap between B W (acc) l and the other precisions is not as pronounced as in the case of ConvNets. This suggests that information is spread equally among all signals which we speculate is due to the shortcut connections preventing the shattering of information BID2. FIG5 indicate that C o leads to convergence and consistently track FL curves with close fidelity. This validates our analysis and justifies the choice of C o. To determine that C o is a close-to-minimal precision assignment, we compare it with: (a) DISPLAYFORM0 is an L × 5 matrix with each entry equal to 1 3, i.e., we perturb C o by 1-b in either direction. FIG5 also contains the convergence curves for the two new configurations. As shown, C −1 always in a noticeable gap compared to C o for both the loss function (except for the CIFAR-10 ResNet) and the test error. Furthermore, C +1 offers no observable improvements over C o (except for the test error of CIFAR-10 ConvNet). These support our contention that C o is close-to-minimal in that increasing the precision above C o leads to diminishing returns while reducing precision below C o leads to a noticeable degradation in accuracy. Additional experimental provided in Appendix D support our contention regarding the near minimality of C o. Furthermore, by studying the impact of quantizing specific tensors we determine that that the accuracy is most sensitive to the precision assigned to weights and activation gradients. We would like to quantify the reduction in training cost and expense in terms of accuracy ing from our proposed method and compare them with those of other methods. Importantly, for a fair comparison, the same network architecture and training procedure are used. We report C W, C A, C M, C C, and test error, for each of the four networks considered for the following training methods:• baseline FL training and FX training using C o, • binarized network (BN) training, where feedforward weights and activations are binary (constrained to ±1) while gradients and accumulators are in floating-point and activation gradients are back- propagated via the straight through estimator BID3 as was done in BID13, • fixed-point training with stochastic quantization (SQ). As was done in BID10, we quantize feedforward weights and activations as well as all gradients, but accumulators are kept in floating-point. The precision configuration (excluding accumulators) is inherited from C o (hence we determine exactly how much stochastic quantization helps), • training with ternarized gradients (TG) as was done in TernGrad . All computations are done in floating-point but weight gradients are ternarized according to the instantaneous tensor spatial standard deviations {−2.5σ, 0, 2.5σ} as was suggested by. To compute costs, we assume all weight gradients use two bits although they are not really fixed-point and do require computation of 32-b floating-point scalars for every tensor. The comparison is presented in TAB3. The first observation is a massive complexity reduction compared to FL. For instance, for the CIFAR-10 ConvNet, the complexity reduction is 2.6× (= 148/56.5), 5.5× (= 9.3/1.7), 7.9× (= 94.4/11.9), and 3.5× (= 49/14) for C W, C A, C M, and C C, respectively. Similar trends are observed for the other four networks. Such complexity reduction comes at the expense of no more than 0.56% increase in test error. For the CIFAR-100 network, the accuracy when training in fixed-point is even better than that of the baseline. The representational and communication costs of BN is significantly greater than that of FX because the gradients and accumulators are kept in full precision, which masks the benefits of binarizing feedforward tensors. However, benefits are noticeable when considering the computational cost which is lowest as binarization eliminates multiplications. Furthermore, binarization causes a severe accuracy drop for the ConvNets but surprisingly not for the ResNets. We speculate that this is due to the high dimensional geometry of ResNets BID1.As for SQ, since C o was inherited, all costs are identical to FX, save for C W which is larger due to full precision accumulators. Furthermore, SQ has a positive effect only on the CIFAR-10 ConvNet where it clearly acted as a regularizer. TG does not provide complexity reductions in terms of representational and computational costs which is expected as it only compresses weight gradients. Additionally, the ing accuracy is slightly worse than that of all other considered schemes, including FX. Naturally, it has the lowest communication cost as weight gradients are quantized to just 2-b. Many works have addressed the general problem of reduced precision/complexity deep learning. Reducing the complexity of inference (forward path): several research efforts have addressed the problem of realizing a DNN's inference path in FX. For instance, the works in address the problem of precision assignment. While Lin et al. FORMULA19 proposed a non-uniform precision assignment using the signal-to-quantization-noise ratio (SQNR) metric, analytically quantified the trade-off between activation and weight precisions while providing minimal precision requirements of the inference path computations that bounds the probability p m of a mismatch between predicted labels of the FX and its FL counterpart. An orthogonal approach which can be applied on top of quantization is pruning BID11. While significant inference efficiency can be achieved, this approach incurs a substantial training overhead. A subset of the FX training problem was addressed in binary weighted neural networks BID5 ) and fully binarized neural networks BID13, where direct training of neural networks with pre-determined precisions in the inference path was explored with the feedback path computations being done in 32-b FL.Reducing the complexity of training (backward path): finite-precision training was explored in BID10 which employed stochastic quantization in order to counter quantization bias accumulation in the weight updates. This was done by quantizing all tensors to 16-b FX, except for the internal accumulators which were stored in a 32-b floating-point format. An important distinction our work makes is the circumvention of the overhead of implementing stochastic quantization BID13. Similarly, DoReFa-Net stores internal weight representations in 32-b FL, but quantizes the remaining tensors more aggressively. Thus arises the need to re-scale and re-compute in floating-point format, which our work avoids. Finally, BID15 suggests a new number format -Flexpoint -and were able to train neural networks using slightly 16-b per tensor element, with 5 shared exponent bits and a per-tensor dynamic range tracking algorithm. Such tracking causes a hardware overhead bypassed by our work since the arithmetic is purely FX. Augmenting Flexpoint with stochastic quantization effectively in WAGE (b), and enables integer quantization of each tensor. As seen above, none of the prior works address the problem of predicting precision requirements of all training signals. Furthermore, the choice of precision is made in an ad-hoc manner. In contrast, we propose a systematic methodology to determine close-to-minimal precision requirements for FX-only training of deep neural networks. In this paper, we have presented a study of precision requirements in a typical back-propagation based training procedure of neural networks. Using a set of quantization criteria, we have presented a precision assignment methodology for which FX training is made statistically similar to the FL baseline, known to converge a priori. We realized FX training of four networks on the CIFAR-10, CIFAR-100, and SVHN datasets and quantified the associated complexity reduction gains in terms costs of training. We also showed that our precision assignment is nearly minimal. The presented work relies on the statistics of all tensors being quantized during training. This necessitates an initial baseline run in floating-point which can be costly. An open problem is to predict a suitable precision configuration by only observing the data statistics and the network architecture. Future work can leverage the analysis presented in this paper to enhance the effectiveness of other network complexity reduction approaches. For instance, weight pruning can be viewed as a coarse quantization process (quantize to zero) and thus can potentially be done in a targeted manner by leveraging the information provided by noise gains. Furthermore, parameter sharing and clustering can be viewed as a form of vector quantization which presents yet another opportunity to leverage our method for complexity reduction. The quantization setup depicted in FIG1 is summarized as follows:• Feedforward computation at layer l: DISPLAYFORM0 where f l is the function implemented at layer l, A l (A l+1) is the activation tensor at layer l (l + 1) quantized to a normalized unsigned fixed-point format with precision B A l (B A l+1), and W l is the weight tensor at layer l quantized to a normalized signed fixed-point format with precision B W l. We further assume the use of a ReLU-like activation function with a clipping level of 2 and a max-norm constraint on the weights which are clipped between [−1, 1] at every iteration.• Back-propagation of activation gradients at layer l: DISPLAYFORM1 where g l (A) is the function that back-propagates the activation gradients at layer l, G DISPLAYFORM2 l+1 ) is the activation gradient tensor at layer l (l + 1) quantized to a signed fixed-point format with precision B G DISPLAYFORM3 ).• Back-propagation of weight gradient tensor G (W) l at layer l: DISPLAYFORM4 where g • Internal weight accumulator update at layer l: DISPLAYFORM5 where U is the update function, γ is the learning rate, and W (acc) l is the internal weight accumulator tensor at layer l quantized to signed fixed-point with precision B W Criterion 1 (EFQN) is used to ensure that all feedforward quantization noise sources contribute equally to the p m budget. Indeed, if one of the 2L reflected quantization noise variances from the feedforward tensors onto p m, say V Wi→pm for i ∈ {1, . . ., L}, largely dominates all others, it would imply that all tensors but W i are overly quantized. It would therefore be necessary to either increase the precision of W i or decrease the precisions of all other tensors. The application of Criterion 1 (EFQN) through the closed form expression in Claim 1 solves this issue avoiding the need for a trial-and-error approach. Because FX numbers require a constant PDR, clipping of gradients is needed since their dynamic range is arbitrary. Ideally, a very small PDR would be preferred in order to obtain quantization steps of small magnitude, and hence less quantization noise. We can draw parallels from signal processing theory, where it is known that for a given quantizer, the signal-to-quantization-noise ratio (SQNR) is equal to SQN R(dB) = 6B + 4.78 − P AR where P AR is the peak-to-average ratio, proportional to the PDR. Thus, we would like to reduce the PDR as much as possible in order to increase the SQNR for a given precision. However, this comes at the risk of overflows (due to clipping). Criterion 2 (GC) addresses this trade-off between quantization noise and overflow errors. Since the back-propagation training procedure is an iterative one, it is important to ensure that any form of bias does not corrupt the weight update accumulation in a positive feedback manner. FX quantization, being a uniform one, is likely to induce such bias when quantized quantities, most notable gradients, are not uniformly distributed. Criterion 3 (RQB) addresses this issue by using η as proxy to this bias accumulation a function of quantization step size and ensuring that its worst case value is small in magnitude. Criterion 4 (BQN) is in fact an extension of Criterion 1 (EFQN), but for the back-propagation phase. Indeed, once the precision (and hence quantization noise) of weight gradients is set as per Criterion 3 (RQB), it is needed to ensure that the quantization noise source at the activation gradients would not contribute more noise to the updates. This criterion sets the quantization step of the activation gradients. Criterion 5 (AS) ties together feedforward and gradient precisions through the weight accumulators. It is required to increment/decrement the feedforward weights whenever the accumulated updates cross-over the weight quantization threshold. This is used to set the PDR of the weight accumulators. Furthermore, since the precision of weight gradients has already been designed to account for quantization noise (through Criteria 2-4), the criterion requires that the accumulators do not cause additional noise. The validity of Claim 1 is derived from the following five lemmas. Note that each lemma addresses the satisfiability of one of the five quantization criteria presented in the main text and corresponds to part of Claim 1. Lemma 1. The EFQN criterion holds if the precisions B W l and B A l are set as follows: DISPLAYFORM0 for l = 1... L, where rnd denotes the rounding operation, B (min) is a reference minimum precision, and E (min) is given by: DISPLAYFORM1 Proof. By definition of the reflected quantization noise variance, the EFQN, by definition, is satisfied if: DISPLAYFORM2 where the quantization noise gains are given by: DISPLAYFORM3 are the soft outputs and ZŶ f l is the soft output corresponding toŶ f l. The expressions for these quantization gains are obtained by linearly expanding (across layers) those used in . Note that a second order upper bound is used as a surrogate expression for p m.From the definition of quantization step size, the above is equivalent to: DISPLAYFORM4 Let E (min) be as defined in FORMULA38: DISPLAYFORM5. We can divide each term by E (min): DISPLAYFORM6 where each term is positive, so that we can take square roots and logarithms such that: DISPLAYFORM7 Thus we equate all of the above to a reference precision B (min) yielding: DISPLAYFORM8 for l = 1... L. Note that because E (min) is the least quantization noise gain, it is equal to one of the above quantization noise gains so that the corresponding precision actually equates B (min). As precisions must be integer valued, each of DISPLAYFORM9 have to be integers, and thus a rounding operation is to be applied on all logarithm terms. Doing so in from Lemma 1 which completes this proof. Lemma 2. The GC criterion holds for β 0 = 5% provided the weight and activation gradients pre-defined dynamic ranges (PDRs) are lower bounded as follows: DISPLAYFORM10 are the largest ever recorded estimates of the weight and activation DISPLAYFORM11, respectively. Proof. Let us consider the case of weight gradients. The GC criterion, by definition requires: DISPLAYFORM12 Typically, weight gradients are obtained by computing the derivatives of a loss function with respect to a mini-batch. By linearity of derivatives, weight gradients are themselves averages of instantaneous derivatives and are hence expected to follow a Gaussian distribution by application of the Central Limit Theorem. Furthermore, the gradient mean was estimated during baseline training and was found to oscillate around zero. DISPLAYFORM13 where we used the fact that a Gaussian distribution is symmetric and Q is the elementary Q-function, which is a decreasing function. Thus, in the worst case, we have: DISPLAYFORM14 Hence, for a PDR as suggested by the lower bound in: DISPLAYFORM15 in Lemma 2, we obtain the upper bound: DISPLAYFORM16 which means the GC criterion holds and completes the proof. For activation gradients, the same reasoning applies, but the choice of a larger PDR in: DISPLAYFORM17 than for weight gradients is due to the fact that the true dynamic range of the activation gradients is larger than the value indicated by the second moment. This stems from the use of activation functions such as ReLU which make the activation gradients sparse. We also recommend increasing the PDR even more when using regularizers that sparsify gradients such as Dropout or Maxout BID8.Lemma 3. The RQB criterion holds for η 0 = 1% provided the weight gradient quantization step size is upper bounded as follows: DISPLAYFORM18 is the smallest ever recorded estimate of σ G DISPLAYFORM19 We close this appendix by discussing the approximation made by invoking the Central Limit Theorem (CLT) in the proofs of Lemmas 2 & 3. This approximation was made because, typically, a backpropagation iteration computes gradients of a loss function being averaged over a mini-batch of samples. By linearity of derivatives, the gradients themselves are averages, which warrants the invocation of the CLT. However, the CLT is an asymptotic which might be imprecise for a finite number of samples. In typical training of neural networks, the number of samples, or mini-batch size, is in the range of hundreds or thousands BID9. It is therefore important to quantify the preciseness, or lack thereof, of the CLT approximation. On way to do so is via the Berry-Essen Theorem which considers the average of n independent, identically distributed random variables with finite absolute third moment ρ and standard deviation σ. The worst case deviation of the cumulative distribution of the true average from the of the approximated Gaussian random variable (via the CLT), also known as the Kolmogorov-Smirnov distance, KS, is upper bounded as follows: KS < Cρ √ nσ 3, where C < 0.4785 . Observe that the quantity ρ σ 3 is data dependent. To estimate this quantity, we performed a forward-backward pass for all training samples at the start of each epoch for our four networks considered. The statistics ρ and σ were estimated by spatial (over tensors) and sample (over training samples) averages. The maximum value of the ratio ρ σ 3 for all gradient tensors was found to be 2.8097. The mini-batch size we used in all our experiments was 256. Hence, we claim that the CLT approximation in Lemmas 2 & 3 is valid in our context up to a worst case Kolmogorov-Smirnov distance of KS < The minimality experiments in the main paper only consider a full 1-b perturbation to the full precision configuration matrix. We further investigate the minimality of C o and its sensitivity to precision perturbation per tenor type. The of this investigation are presented in FIG3. First, we consider random fractional precision perturbations, meaning perturbations to the precision configuration matrix where only a random fraction p of the 5L precision assignments is incremented or decremented. A fractional precision perturbation of 1 (-1) corresponds to C +1 (C −1). A fractional precision perturbation of 0.5 (-0.5) means that a randomly chosen half of the precision assignments is incremented (decremented). FIG3 shows the relative test error deviation compared to the test error associated with C o for various fractional precision perturbations. The error deviation is taken in a relative fashion to account for the variability of the different networks' accuracies. For instance, an absolute 1% difference in accuracy on a network trained on SVHN is significantly more severe than one on a network trained on CIFAR-100. It is observed that for negative precision perturbations the variation in test error is more important than for the case of positive perturbations. This is further encouraging evidence that C o is nearly minimal, in that a negative perturbation causes significant accuracy degradation while a positive one offers diminishing returns. It is also interesting to study which of the 5L tensor types is most sensitive to precision reduction. To do so, we perform a similar experiment whereby we selectively decrement the precision of all tensors belonging to the same type (weights, activations, weight gradients, activation gradients, weight accumulators). The of this experiment are found in FIG3. It is found that the most sensitive tensor types are weights and activation gradients while the least sensitive ones are activations and weight gradients. This is an interesting finding raising further evidence that there exists some form of duality between the forward propagation of activations and back propagation of derivatives as far as numerical precision is concerned. We illustrate a step by step description of the application of our precision assignment methodology to the four networks we report on. Feedforward Precisions: The first step in our methodology consists of setting the feedforward precisions B W l and B A l. As per Claim 1, this requires using. To do so, it is first needed to compute the quantization noise gains using. Using the converged weights from the baseline run we obtain: And therefore, E (min) = 94.7 and the feedforward precisions should be set according to as follows: DISPLAYFORM0 The value of B (min) is swept and p m i evaluated on the validation set. It is found that the smallest value of B (min) ing in p m < 1% is equal to 4 bits. Hence the feedforward precisions are set as follows and as illustrated in gradients. As per Claim 1, an important statistic is the spatial variance of the gradient tensors. We estimate these variances via moving window averages, where at each iteration, the running variance estimateσ 2 is updated using the instantaneous varianceσ 2 as follows: DISPLAYFORM1 where θ is the running average factor, chosen to be 0.1. The running variance estimate of each gradient tensor is dumped every epoch. Using the maximum recorded estimate and we compute the PDRs of the gradient tensors (as a reminder, the PDR is forced to be a power of 2):Layer (min) (min) (min)The value of B (min) is again swept, and it is found that the p m < 1% for B (min) = 3. The feedforward precisions are therefore set as follows and as illustrated in FIG4 Layer Index l 1 2 3 4 5 6 7 8 9 B W l 9 8 9 9 10 9 7 5 5 B A l 8 4 5 4 6 6 7 4 3 Note that for weights, layer depths 21 and 22 correspond to the strided convolutions in the shortcut connections of residual blocks 4 and 7, respectively. The value of B (min) is again swept, and it is found that the p m < 1% for B (min) = 3. The feedforward precisions are therefore set as follows and as illustrated in FIG4 (min)The value of B (min) is again swept, and it is found that the p m < 1% for B (min) = 3. The feedforward precisions are therefore set as follows and as illustrated in

We analyze and determine the precision requirements for training neural networks when all tensors, including back-propagated signals and weight accumulators, are quantized to fixed-point format.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.