Quantization Error as a Metric for Dynamic Precision Scaling in Neural Net Training
Ian Taras Dylan Malone Stuart
cs.LG


Pilot-Streaming: A Stream Processing Framework
  for High-Performance Computing
    Andre Luckow^1,2,3, George Chantzialexiou^1, Shantenu Jha^1,4

   ^1RADICAL, ECE, Rutgers University, Piscataway,NJ 08854, USA

   ^2Clemson University, Clemson, SC 29634, USA

   ^3Ludwig Maximilian University, Munich, Germany

   ^4Brookhaven National Laboratory, Upton, NY, USA
   
    
================================================================================================================================================================================================================================================================================================




Recent work has explored reduced numerical precision for parameters, activations,
and gradients during neural network training as a way to reduce the computational
cost of training <cit.> <cit.>.  We
present a novel dynamic precision scaling (DPS) scheme. Using stochastic fixed-
point rounding, a quantization-error based scaling scheme, and dynamic bit-widths
during training, we achieve 98.8% test accuracy on the MNIST dataset using an
average bit-width of just 16 bits for weights and 14 bits for activations, compared
to the standard 32-bit floating point values used in deep learning frameworks.





§ INTRODUCTION


It is well established that neural networks, though ordinarily trained using 32-bit single precision floating point representation, can achieve desirable accuracy during inference with reduced precision weights and activations  <cit.> <cit.>  <cit.> <cit.>. These reduced precision networks are amenable to acceleration on custom hardware platforms which can take advantage of lower bit-widths in order to speed up computation  <cit.> <cit.>. Reduced precision strategies are not typically applied during back-propagation whilst training, as this can lead to heavily reduced accuracy or even non-convergence.

Recent work has shown that dynamic precision scaling (DPS), a technique in which the numerical precision used during training is varied on-the-fly as training progresses, can achieve computational speedups (on custom hardware) without hampering accuracy  <cit.> <cit.>. DPS uses feedback from the training process to decide on an appropriate number representation. For example, <cit.> suggest starting with reduced precision, and increasing precision dramatically whenever training becomes numerically unstable, or when training loss stagnates.

In this paper, we present a novel DPS algorithm that uses the stochastic fixed-point rounding method suggested by <cit.>, the dynamic bit-width representation used by <cit.>, and an algorithm that leverages information on the quantization error encountered during rounding as a heuristic for scaling the number of fractional bits utilized.



§ FIXED POINT REPRESENTATION AND QUANTIZATION/ROUNDING

Fixed point numbers are represented by a fractional portion appended to an integer portion, with an implied radix point in between. We allow our fixed point representation to use arbitrary bit-width for both the integer and fractional parts, and represent the bit-width of the integer part as IL and the bit-width of the fractional part as FL. We denote a given fixed point representation, then, as ⟨ IL,FL ⟩. DPS modifies IL and FL on-the-fly during training. 

 
Inspired by <cit.>, we use stochastic rounding during quantization of floating point values to ⟨ IL,FL ⟩, as it implements an unbiased rounding.

Our algorithm employs a dynamic bit-width, dynamic radix scheme in which IL and FL are free to vary independently. Note that with the alternative fixed bit-width scheme, IL and FL are inter-dependent as increasing one necessitates a decrease in the other.



§ DYNAMIC PRECISION SCALING ALGORITHM

Here we formally introduce our novel DPS algorithm which leverages average % quantization error as a metric for scaling fractional bits. Quantization error is calculated on a per-value basis as in Equation <ref>. Quantization error % is accumulated and averaged over all round operations – this is the metric used when scaling FL.

    E_% = | x_out - x_in|/x_in× 100

Table <ref> frames this work in relation to prior work in the area.








§ EXPERIMENTS

In order to perform evaluations, we emulate a dynamic fixed point representation by using custom Caffe layers that quantize/round the native floating point values to values that are legal in our fixed point format. In our study, we consider training a neural network using stochastic gradient descent with dynamically scaled precision for weights, activations, and gradients during both the forward (inference) and backward pass. As per <cit.>, we quantize weights, biases, activations, and gradients at the appropriate pass through the network, and update the precision on-the-fly during training on each iteration.

We train LeNet-5 on the MNIST dataset using Caffe and our custom rounding layers and DPS algorithm <cit.>. We use a batch size of 64, and train for 10,000 iterations. We use an initial learning rate of 0.01, momentum of 0.9, a weight decay factor of 0.0005, and scale the learning rate using lr = lr_init * (1 + γ * iter)^-pow, where γ = 0.0001 and pow = 0.75. We update IL and FL once each iteration, and use E_max = R_max = 0.01%.

We compare our results to a baseline network trained on the same dataset with the same hyperparameters, but using full-precision floating point for all attributes. We also compare against a non-dynamic fixed point representation that uses 13 bits for weights and activations, and keeps gradients at 32 bits.



r5.5cm

    < g r a p h i c s >

Moving average bitwidths during training using DPS.


Our results reveal that we can achieve accuracy on-par with the baseline, whilst drastically reducing the bit-width used for both weights and activations. Our dynamic precision scaling algorithm in general, however, doesn't reduce the gradient bit-width very much, as this requires the most precision in order for training to converge. The training loss using DPS is, in general, larger than the training loss of the baseline model without hurting accuracy, suggesting that the reduced precision may act as a regularization technique during training – this needs validation via experimentation on larger networks and more complex datasets. Note that naively reducing the bit-width of weights and activations to a fixed 13-bits with no dynamic precision scaling results in the training process failing to converge. With dynamic precision scaling, however, 13-bit weights and activations are sufficient early in the training process.



§ DISCUSSION

We introduce a dynamic precision scaling algorithm that uses quantization error as a metric for scaling dynamic bit-width fixed point values during neural network training. Combining this with stochastic rounding, we achieve greatly reduced bit-width during training, whilst remaining within a fraction of a % of SOTA accuracy on the MNIST dataset. This avenue of algorithmic work, when paired with emerging hardware for training, has the potential to greatly increase the productivity of engineers and machine learning researchers alike by decreasing training time.


iclr2018_workshop

