Mixed Low-precision Deep Learning Inference using
Dynamic Fixed Point
Naveen Mellempudi1 , Abhisek Kundu1 , Dipankar Das1 , Dheevatsa Mudigere1 , and Bharat Kaul1

arXiv:1701.08978v2 [cs.LG] 1 Feb 2017

1

Parallel Computing Lab, Intel Labs
Bangalore, India

Abstract
We propose a cluster-based quantization method to convert pre-trained full precision weights into ternary weights with minimal impact on the accuracy. In addition
we also constrain the activations to 8-bits thus enabling sub 8-bit full integer inference pipeline. Our method uses smaller clusters of N filters with a common
scaling factor to minimize the quantization loss, while also maximizing the number
of ternary operations. We show that with cluster size of N=4 on Resnet-101, can
achieve 71.8% TOP-1 accuracy, within 6% of the best full precision result, while
replacing ≈ 85% of all multiplications with 8-bit accumulations. Using the same
method with 4-bit weights achieves 76.3% TOP-1 accuracy which within 2% of
the full precision result. We also study the impact of the size of the cluster on both
performance and accuracy, larger cluster sizes N=64 can replace ≈ 98% of the
multiplications with ternary operations but introduces significant drop in accuracy
which necessitates fine tuning the parameters with retraining the network at lower
precision. To address this we have also trained low-precision Resnet-50 with 8-bit
activations and ternary weights by pre-initializing the network with full precision
weights and achieve 68.9% TOP-1 accuracy within 4 additional epochs. Our final
quantized model can run on a full 8-bit compute pipeline, with a potential 16x
improvement in performance compared to baseline full-precision models.

1

Introduction

Deep Learning has achieved unparalleled success with large-scale machine learning. Deep Learning
models are used for achieving state-of-the-art results on a wide variety of tasks including Computer
Vision, Natural Language Processing, Automatic Speech Recognition and Reinforcement Learning
[1]. Mathematically this involves solving a complex non-convex optimization problem with order
of millions or more parameters. Solving this optimization problem - also referred to as training the
neural network is a compute-intensive process that for current state-of-art networks requires days
to weeks. Once trained, the DNN is used by evaluating this many-parameter function on specific
input data - usually referred to as inference. While the compute intensity for inference is much lower
than that of training, inference also involves significant amount of compute. Moreover, owing to the
fact that inference is done on a large number of input data, the total computing resources spent on
inference is likely to dwarf those that are spent on training. Due to the large and somewhat unique
compute requirements for both deep learning training and inference operations, it motivates the use of
non-standard customized arithmetic [6, 2, 5, 14, 8, 7] and specialized compute hardware to run these
computations as efficiently as possible [4, 15, 13, 11]. Furthermore, there some theoretical evidence
and numerous empirical observations that deep learning operations can be successfully done with
much lower precision.
In this work we focus on reducing the compute requirements for deep learning inference, by directly
quantizing pre-trained models with minimum (or no) retraining and achieve near state-of-art accuracy.

Our paper makes the following contributions:
1. We propose a novel cluster-based quantization method to convert pre-trained weights to
lower precision representation with minimal loss in test accuracy.
2. On Resnet-101 with 8-bit activations and using cluster size (N=4) to quantize weights, we
achieve 76.3% TOP-1 accuracy with 4-bit weights and 71.8% TOP-1 accuracy with 2-bit
ternary weights. To the best of our knowledge this is the best reported accuracy with ternary
weights on ImageNet dataset[3], without retraining the network.
3. We explore the performance-accuracy trade-off using different cluster sizes with ternary
weight representation. For a cluster size of N, we reduce the higher precision ops (8-bit
multiply) to one for every N ∗ K 2 lower precision ops (8-bit accumulation), which results
significant reduction in computation complexity. Using smaller cluster size of N=4 we
achieve state-of-the-accuracy, but larger cluster sizes (N=64) would require retraining the
network at lower precision to achieve comparable accuracy.
4. We train a pre-initialized low precision Resnet-50 using 8-bit activations and 2-bit weights
using larger cluster (N=64) and achieve 68.9% TOP-1 accuracy on ImageNet dataset[3]
within 4-epochs of fine-tuning.

2

Related Work

Deep learning training and inferencing are highly compute intensive operations, however using full
precision (F P 32) computations on conventional hardware is inefficient and not strictly warranted
from functional point-of-view. To address this issue, there has been a lot of interest at using lower
precision for deep learning, in an attempt to identify the minimum required precision to ensure
functional correctness within acceptable thresholds.
In the past many researches have proposed low-precision alternatives to perform deep learning tasks.
Vanhoucke et al.[12] showed that using 8-bit fixed-point arithmetic convolution networks can be sped
up by up to 10x on speech recognition tasks on general purpose CPU hardware. Gupta et al.[4] have
successfully trained networks using 16-bit fixed point on custom hardware. Miyashita et al.[8] used
log quantization on pre-trained models and achieved good accuracy by tuning the bit length for each
layer. More recently, Venkatesh et al.[13] achieved near state of the art results using 32b activations
with 2-bit ternary weights on Imagenet dataset. Hubara et al.[5] have demonstrated that with weights
as binary values training from scratch can achieve near state-of-the-art results for ILSVRC 2012
image classification task[9].

3

Low Precision Inference

In this paper, we primarily focus on improving the performance and accuracy of the inference task. We
explore possibility of achieving high accuracy using sub 8-bit precision on state-of-the-art networks
without expensive retraining. Previous work from Miyashita et al.[8] showed that by compressing
the dynamic range of the input, it is possible to minimize the quantization loss and achieve high
accuracy. We take a different approach to minimize the impact of dynamic range on quantization. We
propose a cluster-based quantization method that groups weights into smaller clusters and quantize
each cluster with a unique scaling factor. We use static clustering to group filters that accumulate to
the same output feature to simplify the convolution operations. Empirical evidence also suggests that
these clusters which learn similar features tend to have smaller dynamic range. Using dynamic fixed
point representation, this method can effectively minimize the quantization errors and improve the
inference accuracy of quantized networks. Applying this scheme on a pre-trained Resnet-101 model,
with 4-bit weights and 8-bit activations, we achieve 76.3% TOP-1 accuracy on ImageNet dataset[3],
without any retraining.
3.1

2-bit Ternary Weights

Going below 4-bits we use the the ternary representation for weights, following the threshold based
approximation proposed by Li et al [7], i.e., approximate full-precision weight W ≈ αŴ in `2 norm
2

Algorithm 1 Ternarize Weights
1: Input: Learned full-precision weights W of a layer with d filters.
2: Group filters into k clusters: {Gj }, j = 1, ..., k. Let N = |Gj | (number of filters in Gj ).
3: For each cluster Gj
4:
Run Algorithm 2 on each filter W ∈ Gj , and store the thresholds as a vector α.
5:
For t = 1, ...,
qN , Tt = {i : αi belongs to the top t elements of sorted α}.
6:

P

Set αt =

i∈Tt

αi2 /|Tt |.

(t)

7:
Construct Ŵ(t) , such that, Ŵi = Sign(Wi ), if |Wi | > αt , and 0 otherwise.
P
8:
Find αt∗ and Ŵ∗(t) that minimizes W∈Gj kW − αt Ŵ(t) k2F .
9:
Let α̂t∗ be a reduced-precision representation of αt∗ .
10: Output: k number of α̂t∗ and the group of ternary weights Ŵ.

Algorithm 2 Threshold Selection
1: Input: W ∈ Rn .
2: Sort elements of W according to magnitude.
3: For τ ∈ [0, 1], Iτ = {i : |Wi | belongs to the top b τ · n c elements of sorted list}.
(τ )

4: Construct Ŵ(τ ) , such that, Ŵi
5: Set ατ =

= Sign(Wi ), for i ∈ Iτ , and 0 otherwise.

qP

2
i∈Iτ Wi /|Iτ |.

6: Compute ατ ∗ that minimizes kW − ατ Ŵ(τ ) k2F , for τ ∈ [0, 1].
7: Output: ατ ∗

k · kF , where α is a scaling factor and Ŵ is a ternary weight with Ŵi ∈ {−1, 0, +1}. Where W is
the matrix representing learned full-precision weights, and Ŵ represents the corresponding ternary
representation. We apply the block-quantization method described in section-3, to compute multiple
scaling factors for each layer to minimize the accuracy loss. Our method differs from [7] in the
approximation used for computing scaling factor (α). We use the RM S formulation as shown by the
equation (1). The intuition behind using RM S term is to push the threshold parameter towards larger
values within the cluster which helps speed up weight pruning.
sP
α=

W2i
, where |Iτ | is the number of elements in Iτ .
|Iτ |

i∈Iτ

(1)

In addition, we run our search algorithm1 in hierarchical fashion by minimizing the error within each
filter first and then within the cluster of filters. Experimental evidence shows that these improvements
help finding the optimal scaling factor that minimizes quantization loss.
Using multiple scaling factors can lead to more 8-bit multiplications. Hence, we choose the cluster
size carefully to improve the ratio of low-precision (2-bit) to high-precision (8-bit) operations (Section
3.3). Our algorithm (Algorithm 1) takes the full-precision learned weights and returns clusters of
ternary representation of groups of kernels along with their scaling factors. We further quantize the
scaling factors down to 8-bit to eliminate any operation that requires more than 8 bits. Applying this
scheme on pre-trained ResNet-101 model, using 8-bit activations we achieve 71.8% TOP-1 accuracy
on ImageNet dataset.
3.2

C1 and BatchNorm Layers

In our experiments we keep weights of the first convolution layers at 8-bits to prevent from accumulating losses while the rest of the layers including fully connected layers operate at lower precision.
We also recompute the batch norm parameters during the inference phase to compensate for the
shift in variance introduced by quantization. This is essential for making it work, when we are not
retraining at lower precision. We are exploring the possibility of fusing batch normalization layers
with the convolution layers before quantization to avoid this extra computation.
3

Figure 1: Resnet-101 results on ImageNet dataset using 8-bit activations with 4-bit weights 8a − 4w
and 2-bit weights 8a − 2w

3.3

Performance Implications

Choosing the right cluster size is a trade-off between performance and accuracy, while having one
cluster per layer favors higher compute density by eliminating all multiplications, it’s not ideal for
achieving high accuracy. Although, previous research in this space[13] showed that it is possible
to recover some of this lost accuracy through retraining. It’s not always ideal, because of the costs
involved in retraining these networks in low-precision, not to mention the technical difficulties
involved in achieving reasonable solution on these networks.
We explored the accuracy-performance trade-off with various cluster sizes. Our experiments show
on Resnet-101, using a cluster size of N = 4 can achieve 71.8% TOP-1 accuracy, within 6% of
the full precision result. This result significant because this is to the best of our knowledge highest
accuracy achieved on Imagenet dataset[3] without retraining the network in low-precision. In terms
of performance impact, the clustering will result in one 8-bit multiplication for the entire cluster
(N ∗ K 2 ) of ternary accumulations. Assuming roughly 50% of the convolutions are 3x3 and the
rest are 1x1, using this block size can potentially replace 85% of multiplications in Resnet-101
convolution layers with simple 8-bit accumulations. For networks that predominantly use filters that
are 3x3 or bigger, this ratio would be greater than 95%. We explored the accuracy-performance
trade-off with various cluster sizes, we concluded that using cluster size of N = 64, we can replace
≈ 98% of multiplications in Resnet-101 with 8-bit accumulations, but with a significant loss to the
accuracy. At this point retraining the network at lower precision would be necessary.

4

Training with Low-precision

We trained the low precision ResNet-50 on ImageNet dataset using 2-bit weights and 8-bit activations
by initializing the network with pre-trained full precision model. We take the approach proposed
by Marcel et al.[10], and replace data pre-processing steps such as mean-subtraction and jittering
with batch normalization layer inserted right after the data later. We obtained the pre-trained models
published by Marcel et al.[10] and fine-tune the parameters of our low-precision network. In the
forward pass, the weights are converted to 2-bit ternary values using the algorithm described in 1
in all convolution layers, except the first layer, where the weights are quantized to 8-bit fixed point
representation. Activations are quantized to 8-bit fixed point in all layers including ReLU, BatchNorm
layers. We did not quantize the weights in FC layer for the training exercise. Gradient updates are
performed in full precision for convolution and FC layers. We reduced the learning rate to an order of
1e-4, in order to avoid exploding gradients problem, while we keep all the other hyper parameters
same as that of full precision training. After running for 4-epochs, we recovered most of the accuracy
4

and achieved 68.6% Top-1 and 88.7% Top-5 accuracy compared to our baseline 75.02%(Top-1),
92.2%(Top-5).

Figure 2: Fine-tuning Resnet-50 with pre-initialized weights on Imagenet dataset.

5

Conclusion

We propose a clustering based quantization method which exploits local correlations in dynamic
range of the parameters to minimize the impact of quantization on overall accuracy. We demonstrate
near SOTA accuracy on Imagenet data-set using pre-trained models with quantized networks without
any low precision training. On Resnet-101 using 8-bit activations the error from the best published
full precision (F P 32) result is within ≈ 6% for ternary weights and within ≈ 2% for 4-bit weights.
To the best of our knowledge this is the best achieved accuracy with ternary weights for Imagenet
dataset.
Our clustering based approach allows for tailoring solutions for specific hardware, based on the
accuracy and performance requirements. Smaller cluster sizes achieves best accuracy, with N=4
≈ 85% of the computations as low precision operations (simple 8-bit accumulations) and this is
better suited for implementation on specialized hardware. Larger cluster sizes are more suited to
current general purpose hardware, with a larger portion of computations as low precision operations
(> 98% for N=64), however this comes with the cost of reduced accuracy. This gap can be bridged
with additional low precision training as show in section 4, work is underway to further improve
this accuracy. Our final quantized model can be efficiently run on full 8-bit compute pipeline, thus
offering a potential 16X performance-power benefit.
Furthermore as continuation of this work, we are looking in a more theoretical exploration to better
understand the formal relationship between the clustering and final accuracy, with an attempt establish
realistic bounds for given network-performance-accuracy requirement.

5

References
[1] Yoshua Bengio, Ian Goodfellow, and Aaron Courville. Deep learning. Book in preparation for
MIT Press, 2016.
[2] Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized
neural networks: Training deep neural networks with weights and activations constrained to+ 1
or-1. arXiv preprint arXiv:1602.02830, 2016.
[3] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale
hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009.
IEEE Conference on, pages 248–255. IEEE, 2009.
[4] Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning
with limited numerical precision. In ICML, pages 1737–1746, 2015.
[5] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized
neural networks. In Advances in Neural Information Processing Systems, pages 4107–4115,
2016.
[6] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations.
arXiv preprint arXiv:1609.07061, 2016.
[7] Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. arXiv preprint arXiv:1605.04711,
2016.
[8] Daisuke Miyashita, Edward H Lee, and Boris Murmann. Convolutional neural networks using
logarithmic data representation. arXiv preprint arXiv:1603.01025, 2016.
[9] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng
Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual
recognition challenge. International Journal of Computer Vision, 115(3):211–252, 2015.
[10] Marcel Simon, Erik Rodner, and Joachim Denzler. Imagenet pre-trained models with batch
normalization. arXiv preprint arXiv:1612.01452v2, 2016.
[11] Yaman Umuroglu, Nicholas J Fraser, Giulio Gambardella, Michaela Blott, Philip Leong,
Magnus Jahre, and Kees Vissers. Finn: A framework for fast, scalable binarized neural network
inference. arXiv preprint arXiv:1612.07119, 2016.
[12] Vincent Vanhoucke, Andrew Senior, and Mark Z Mao. Improving the speed of neural networks
on cpus. In Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop, volume 1,
page 4. Citeseer, 2011.
[13] Ganesh Venkatesh, Eriko Nurvitadhi, and Debbie Marr. Accelerating deep convolutional
networks using low-precision and sparsity. arXiv preprint arXiv:1610.00324, 2016.
[14] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net:
Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint
arXiv:1606.06160, 2016.
[15] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv
preprint arXiv:1612.01064, 2016.

6

