Finite Precision Error Analysis of Neural Network Hardware Implementations.
Through parallel processing, low precision fixed point hardware can be used to build a very high speed neural network computing engine where the low precision results in a drastic reduction in system cost. The reduced silicon area required to implement a single processing unit is taken advantage of by implementing multiple processing units on a single piece of silicon and operating them in parallel. The important question which arises is how much precision is required to implement neural network algorithms on this low precision hardware. A theoretical analysis of error due to finite precision computation was undertaken to determine the necessary precision for successful forward retrieving and back-propagation learning in a multilayer perceptron. This analysis can easily be further extended to provide a general finite precision analysis technique by which most neural network algorithms under any set of hardware constraints may be evaluated.
Introduction
The high speed desired in the implementation of many neural network algorithms, such as
back-propagation learning in a multilayer perceptron (MLP), may be attained through the
use of finite precision hardware. This finite precision hardware; however, is prone to errors.
A method of theoretically deriving and statistically evaluating this error is presented and
could be used as a guide to the details of hardware design and algorithm implementation.
The paper is devoted to the derivation of the techniques involved as well as the details of
the back-propagation example. The intent is to provide a general framework by which most
neural network algorithms under any set of hardware constraints may be evaluated.
Section 2 demonstrates the sources of error due to finite precision computation and
their statistical properties. A general error model is also derived by which an equation for
the error at the output of a general compound operator may be written. As an example,
error equations are derived in Section 3 for each of the operations required in the forward
retrieving and error back-propagation steps of an MLP. Statistical analysis and simulation
results of the resulting distribution of errors for each individual step of an MLP are also
included in this section. These error equations are then integrated, in Section 4, to predict
the influence of finite precision computation on several stages (early, middle, final stages)
of back-propagation learning. Finally, concluding remarks are given in Section 5.
2 Sources of Error in Finite Precision Computation
For a finite precision computation of a nonlinear operation of multiple variables, several
sources of error exist. For example, in the computation of y = OE(wx), the two input
variables, w and x, have input errors ffl w and ffl x , respectively, whose sources are prior
finite precision data manipulations. There are errors generated due to the finite precision
computation of involved operators. More specifically, the finite precision multiplication of
the two variables generates one error, ffl 3 . Similarly, the finite precision nonlinear operator
OE generates the other error, ffl OE . Therefore, the resulting finite precision result ~
y is equal to
~
where we assume that the error product ffl w ffl x is negligible, and a first order Taylor series
approximation is used.
The input errors are propagated through the operators. For example, the multiplication
of the two variables with finite precision errors propagates error, i.e., wffl x This
propagated error along with the generated finite precision multiplication error, ffl 3 , further
propagates through the nonlinear operator, resulting in the total finite precision error,
The total finite precision error ffl y imposed on y will then become the input finite precision
error of variable y for future operations.
2.1 Error Generation and Propagation by Successive Operators
A compound operator, which is produced by successive operators OE
is shown in Figure (1). Error at the input, ffl x , and error generated in each operator, ffl OE i
is propagated through the remaining operators to the output. We can approximate the
output error, ffl y , in terms of ffl x , ffl OE i
, and OE i [8]. From Figure (1),

Figure

1: Successive operators generating and propagating error, where
If y i is defined as the intermediate result after the first i successive operators,
and
. (5)
Carrying out similar expansion for all intermediate values, we can rewrite ~ y to be
Y
where
k (1) is defined to be 1.
The product shown is just the chain rule for derivative, @y
, which can be further approximated
by the derivative without error, @y
. Note that this approximation is equivalent
to the approximation already made in the first order Taylor series.
Y
Y
Therefore,
@y
@x
@y
2.2 Error Generation and Propagation by General Compound Operators
The effects of finite precision error at the output of a general system of compound operators
with multiple input variables can be calculated through an extension of the previous analysis
for successive operators of a single variable [8]. The following steps are employed:
1. Break the computation into a calculation graph (see the example given in Figure (2)).
The general calculation graph is made up of n operators, fOE i g, and m system inputs,
g.
2. Number the operators, OE , such that the intermediate generated inputs,
fy k g, to an operator, OE i , have lower indices than the operator output, y i . By extending
Eq. (8) to multiple inputs, the total finite precision error, ffl y , is given as [8]:
@y
@y
3. Using the calculation graph, the partial derivatives,
and
, are evaluated
and substituted into Eq. (9) to give an equation for ffl y .
4. Statistical methods discussed below are then used to evaluate the error in Eq. (9).
Methods include the computation of mean and variance for various functions of random
variables, as well as approximations using the central limit theorem.
2.3 Common Techniques of Finite Precision Computations
Three common techniques are used in finite precision computations: truncation, jamming,
and rounding. The truncation operator simply chops the q lowest order bits off of a number
and leaves the new lowest order bit, in the 2 r -th place, unchanged. The jamming operator
chops off the q lowest order bits of a number and forces the new lowest order bit, in the 2 r -th
place, to be a "1" if any of the q removed bits were a "1"; otherwise, the new lowest order
bit retains its value. This operation is equivalent to replacing the 2 r -th bit with the logical
OR of the 2 r -th bit and the q bits which are chopped off. The jamming operator has the
advantage of generating error with zero mean, but generates error with a higher variance
compared with the truncated one. The rounding operator also chops off the q lowest order
bits of a number which will have its new lowest order bit in the 2 r -th place. If the q bit
value chopped off is greater than or equal to 2 r01 , the resulting value is incremented by 2 r ;
otherwise, it remains unchanged [1].
The error generated by truncating, jamming, or rounding techniques may be considered
to be a discrete random variable distributed over a range determined by the specific
technique being employed. For a statistical view of the error, it is desirable to know the
mean and variance of the error generated by each of these three techniques. For a discrete
random variable, x, the mean is given by:
and the variance is:
is usually assumed to be uniformly distributed.
Truncation: Truncation generates error which is uniformly distributed in the range [02 r
with each of the 2 q possible error values being of equal probability. Therefore,
1. Mean and variance may then be computed.
Jamming: The error generated by jamming is not uniformly distributed as the probability
of the error being zero is twice the probability of the error holding any of its other possible
values. The range of error is
and x This results in 2 q+1 0 1 possible error values.
The mean and variance of jamming error are then
Rounding: Rounding generates error which is uniformly distributed in the range [02 r01 ,
with each of the 2 q possible error values being of equal probability. Therefore,
1. The mean and variance are computed.
Nonlinear Functions of Discrete Random Variables: For a nonlinear function of a
discrete random variable, x, the mean and variance are given by:
2.4 Statistical Properties of Independent Random Variables
For two independent random variables, x and y, with means - x ; - y and variances oe 2
y ,
and a constant a, the following properties of mean and variance can be shown [7].
1. Multiplication by a constant.
2. Sum of two independent random variables.
3. Product of two independent random variables.
2.5 Statistical Properties of Sum of Independent Random Variables
Expected Squared Error: The expected squared error can be written in terms of the
mean and variance of the error. Consider a set of errors which are independent random
variables, fffl i g, with mean - and variance oe 2 . Then, the expected value of the average sum
of the squares of fffl i g can be written,
"N
Noting that oe
the expected value of the average sum squared errors is
equal
Central Limit Theorem: The central limit theorem [7] states that if fx i g are independent
random variables, then the density of their sum, properly normalized, tends to
a normal curve as N ! 1. For discrete random variables, the probabilities tend to the
samples of a normal curve. Normalization can be achieved in a couple of different ways. If
are discrete random variables with mean - and variance oe 2 , then the sum x;
results in -
Invoking the central limit theorem, the probability that the sum of the random variables,
x, is equal to the discrete value x k which approaches the sample of a normal curve when N
is large.
oe x
Sum of Products of Independent Random Variables: The central limit theorem
can be extended to cover the case where the random variable being summed is a product
of random variables. Say that for independent random variables fx i g and fy i g,
then the probability density of the random variable xy approaches a normal curve for large
N with mean and variance equal to
xy (28)
3 Application to Neural Network Retrieving and Learning
It has been shown that operations in both the retrieving and the learning phases of most of
the neural network models can be formulated as a linear affine transformation interleaved
with simple linear or nonlinear scalar operation. In terms of the hardware implementations,
all these formulations call for a MAC (multiply and accumulation) processor hardware
[4, 5, 1]. Without loss of generality, we will specifically discuss the multilayer perceptron
neural network model and the back-propagation learning [10, 9].
3.1 Forward Retrieving and Back Propagation of an MLP
Given a trained (fixed-weight) MLP, the retrieving phase receives the input test pattern,
fx 0;i g, and propagates forward through the network to compute the activation values of
the output layer, fx L;j g, which will be used as the indicator for classification or regression
purposes. On the other hand, in the commonly used learning phase of an MLP, the input
training pattern, fx 0;i g, is first propagated forward through the network and the activation
values, fx l;j g, are computed according to the same forward operations used in the retrieving
phase. Then the output activation values, fx L;j g, are compared with the target values
g, and the value of the output delta, fffi L;j g, for each neuron in the output layer is
derived. These error signals are propagated backward to allow the recursive computation
of the hidden delta's, fffi l;j g, as well as the update values of the weights, f1w l;i;j g, at each
layer. While many methods have been proposed to accelerate learning by estimating the
local curvature of the training error surface using second order derivative information, the
discussion of these methods are beyond the scope of this paper, and can be referred to [3].
The operations in the forward retrieving of an L-layer perceptron can be formulated as
a forward affine transformation interleaved with a nonlinear scalar activation function:
where x l;i denotes the activation value of the i th neuron at the l th layer, w l+1;i;j denotes the
synaptic weight interconnecting the i th neuron at the l th layer and the j th neuron at the
th layer. The nonlinear activation function, f(1), is usually taken to be sigmoidal.
The learning of an MLP follows the iterative gradient descent approach with the following
update at each presentation of a training data pair [10, 9]:
where the computation of the back-propagated error, ffi l;j , can again be formulated as a
backward affine transformation interleaved with the post-multiplication of the derivatives
of the nonlinear activation function:
with initial output-layer propagated error being
3.2 Finite Precision Analysis of Forward Retrieving
Explicitly following the procedure discussed in Section 2.2, the calculation graph of the
forward retrieving operation, with simplified notation (see Eq. (29)), in an MLP is shown
in

Figure

@ @R @ @R09 0x n w n;j
f y

Figure

2: Calculation graph for the forward retrieving, of an MLP, where 3
denotes a truncation, jamming, or rounding operator.
To carry out the analytical formula as given in Eq. (9) for the forward retrieving of an
MLP, several partial derivatives need to be computed from Eq. (33):
@y
@w i;j
@y
@y i3
@y i3
@y i+
@y i+
@y
By substituting the values for the partial derivatives in Eqs. (34) to (37) and the
generated and propagated errors for variables and operators into Eq. (9),
ffl y i3
3.3 Finite Precision Analysis of Output
From Eq. (32), the calculation graph for the computation of back-propagated error in an
output neuron, with simplified notation, is shown in Figure (3).
@ @R 00
@ @R 02&3
y j0

Figure

3: Calculation graph for in an output neuron.
Again, to carry out the analytical formula as given in Eq. (9) for the output delta
computation of an MLP, the partial derivatives of Eq. (39) are evaluated.
@y
@y
@y
@y j0
Substituting for the partial derivatives and individual error terms, the overall finite precision
error for the output delta computation is:
3.4 Finite Precision Analysis of Hidden
From Eq. (31), the calculation graph for the computation of back-propagated error in a
hidden layer neuron, with simplified notation, is shown in Figure (4).
Following similar partial derivative evaluations using Eq. (45), we can again compute
the finite precision error for the hidden delta (see Eq. (9)):
ffl y k3
ffl y k+
@ @R 02&3

Figure

4: Calculation graph for in a hidden neuron.
3.5 Finite Precision Analysis of Weight Update
From Eq. (30), the calculation graph for the computation of the weight update (without
the momentum term), with simplified notation, is shown in Figure (5).
@ @R 02&3
y j3

Figure

5: Calculation graph for
Following similar partial derivative evaluations using Eq. (47), we can again compute
the finite precision error for weight update (see Eq. (9)):
3.6 Statistical Evaluation of the Finite Precision Errors
Given the analytical expressions of all the finite precision errors associated with the forward
retrieving and back-propagation of an MLP, a statistical evaluation of these errors is
undertaken. This evaluation is based on the mean and variance analysis using truncation,
jamming, and rounding techniques, and also based on statistical properties of independent
random variables, sums of independent random variables, and sums of products of independent
random variables discussed in Sections 2.3, 2.4, and 2.5. The first step is to choose
the precision of each component which will be employed at each step in the problem. A
practical limited precision implementation of the MLP algorithm might use precisions as
follows [2].
1. All neurons have 8-bit (8 bits to the right of the decimal) inputs, outputs, and targets
with range [0:0; 1:0).
2. All weights and biases use 16 bits (one sign bit, 3 bits to the left and 12 bits to the
right of the decimal) with range [08:0; 8:0).
3. The output ffi is 8 bits (one sign bit and 7 bits all to the right of the decimal) with
range [00:5; 0:5).
4. Hidden ffi uses 16 bits (one sign bit, 3 bits to the left and 12 bits to the right of the
decimal) with range [08:0; 8:0).
Finite Precision Error in Forward Retrieving: The expected forward retrieving error
can be calculated for both a single layer of neurons and for multiple layers of neurons by
propagating upward the finite precision errors of the lower layers. First, a simplification may
be made to Equation (38). The multiply and accumulate steps can be computed without
generating any error if enough bits (e.g., 24 bits) are used for each of the intermediate steps,
y i3 , and y i+ . In this case, ffl y i3
This is practical since the expense of accumulator
precision is very small.
The 3 operator now reduces the final 24-bit sum to an 8-bit (one sign bit, 3 bits to the
left and 4 bits to the right of the decimal) value which is used as the input for the sigmoid
look-up table. Equation (38) may now be rewritten as
Before invoking the central limit theorem on the sums, it is necessary to know the
distributions of the random variables w ij , ffl w ij
.

Table

1 shows the
the statistically evaluated values for each contributing component of Eq. (49) based on
the assumptions of bit sizes given above. The evaluation starts with 16-bit weights which
are uniformly distributed across the entire range, [08; 8), and weight error comes from
truncating 24-bit weights to 16 bits. For an input neuron, x i is an 8-bit value uniformly
distributed over [0:0; 1:0) which has been truncated from a 24-bit value. Therefore, ffl x i
is
the truncation error with 16. The distribution of f 0
j is approximated as
a function of a normally distributed random variable. ffl 3 is the error generated when the
accumulated 24-bit value is jammed to become an 8-bit value, and ffl f j
is the error generated
in the lookup table which is approximated to be the same as rounding 2 bits at the
place.
R.V. Type q r - oe 2
rounding 8 -12
truncation
jamming
rounding 2 -5

Table

1: Mean and variance of variables in the forward retrieving calculation.
Based on the precision assumptions given in Table (1) with various sizes of bit-allocation
to the weights, Figure (6) shows the statistically evaluated average sum of the squares of
ffl y , as defined in Eqs. (49) and (24), due to the finite precision computation of a single step
forward retrieving. In these evaluations, for any weight bit number (say k-bits), the weight
always contains one sign bit, 3 left-of-decimal bits, and k04 right-of-decimal bits. The lower
solid curve shows the statistical evaluation of finite precision error introduced in neurons
of the first hidden layer and the upper solid curve shows that of the second hidden layer
(or the output layer in a 2-layer perceptron). Note that the statistical evaluation of errors
shows a dive-in at around 8-bit weights. This fact suggests that for the implementation of
finite precision hardware for only the forward retrieving purpose, we can train the network
using high precision computation (under the constraint of only sign plus 3 bits to the left
of the decimal), and then download the well trained finite precision portion (8 bits total,
or 4 bits to the right of the decimal) of the weights into the hardware. The performance
degradations due to this finite precision conversion is almost negligible.
Weight Bits
Average
Squared

Figure

Statistically evaluated and simulation evaluated values of E[ffl 2
introduced to
neurons of the first and second hidden layers.
Simulations are also conducted to verify the statistical evaluations. A 2-layer perceptron
with 100 inputs, 100 hidden neurons and 100 output neurons is simulated. 100 sets of
randomly generated 100-dimensional input data are tested and averaged. All the weights of
the network are also randomly generated. The average sum of the squares of the differences
between the finite precision computation and the full precision (64 bits for all the contributing
components) computation can thus be obtained. The lower dashed curve shows
the simulated evaluation of finite precision error introduced in neurons of the first hidden
layer, and the upper dashed curve shows that of the output layer. Both curves match quite
consistently with those produced by statistical evaluations.
4 Finite Precision Analysis for Iterative Learning
As discussed in Section 3.1, back-propagation learning involves all four consecutive steps
of computations: forward retrieving, output delta computation, hidden delta computation,
and weight updating. Therefore, for each weight updating iteration at the presentation of
any training pattern, the finite precision errors ffl 1w introduced to f1w l;i;j g given in Eq.
(48) is in fact a propagated result of the error generated from the previous three steps.
Therefore, the final mathematical expression of finite precision error for a single weight
updating iteration can be formulated in a straightforward manner based on the existing
derivations given in Eqs. (38), (44), (46), and (48). The statistical evaluation value for
the average sum of the squares of weight updating error ffl 1w due to the finite precision
computation of a single learning iteration can thus be computed.
The back-propagation learning discussed above is simply a nonlinear optimization problem
based on simple gradient descent search, with an elegant way of computing the gradients
using the chain rule on the layers of the network. This gradient descent search updates the
weights based only on the first derivative approximation of the error surface with the up-dating
of each individual weight being independent of the others [6]. Therefore, even if the
approach is computationally efficient, it can behave very unwisely and converge very slowly
for complex error surfaces. Due to the strong influence introduced in the gradient descent
search approximation, the real effect to the learning convergence and accuracy due to finite
precision computation will be difficult to measure. Therefore, the statistically evaluated
average sum of the squares of ffl 1w , by itself, can not determine a network's propensity to
learn.
4.1 Ratio of Finite Precision Errors
A more meaningful measure, ae, which potentially indicates the effect of finite precision
computation to the weight updating, can be defined as the ratio of the statistical average
sum of the squares of finite precision weight updating error ffl 1w and that of full precision
weight updating magnitude 1w 2
This ratio serves as a useful indicator of the additional impact caused by finite precision
computation on top of that caused by the gradient descent approximation of back-propagation
learning. The ratio depends not only on the number of bits assigned to the finite
precision computation, but on the current stage of learning progress, which can be specified
by the distribution of the difference between the desired and actual output g. More
specifically,
where we assume that ffl x L;j
0, since the ability to learn should depend on the ability
to learn these finite precision values.
Based on the same practical choices of finite precision bit size given in Section 3.6 vs.
the number of bits (say k bits) assigned to the weights fw ij g and weight updates f1w ij g,
we can statistically evaluate this ratio at several different stages of learning. Figure 7 shows
the statistical evaluation values of the finite precision ratio for the weights connecting the
neurons between the hidden and the output layers in a 2-layer MLP. 4 different values of
are used, which represent four different stages of learning: early stage
middle stage convergence stage convergence
stage Figure 8 shows the statistical evaluation values of the finite precision
ratio for the weights connecting the neurons between the hidden and the input layers for
these four different values of fi.
Note that the finite precision ratio curves gradually (along various stages of learning)
dive around the region where the number of bits for weights are 12-14 bits for soft convergence
stage, 14-16 for hard convergence stage of learning, and then get almost steady
when the number of bits is increased further. This dive point indicates the potential strong
disturbance to the convergence and accuracy of back-propagation leaning in the long run.
Therefore, it gives a good guideline to which number of bits are required for the weights
in the learning so as to have similar learning convergence and accuracy as that attained
using high precision computation. It is also interesting to note that at the later stage of the
learning the impact of the finite precision error is getting larger due to the smaller values
of fi when the network is fine-tuning its weights.
Weight Bits
Finite
Precision

Figure

7: The statistical evaluation values of the finite precision ratio for the top-layer
weights in a 2-layer MLP. Four different stages of learning are evaluated.
4.2 Simulation Results for Iterative Learning
To verify the theoretical evaluation of the impact caused by the finite precision computation,
a simple regression problem is designed, which maps 2-dimensional inputs, fx 1
1-dimensional outputs, fyg:
An MLP containing 2 input neurons, 8 hidden neurons, and 1 output neuron, is adopted.
There are 256 pairs of randomly selected data for training. Finite precision learning simu-
Weight Bits
Finite
Precision

Figure

8: The statistical evaluation values of the finite precision ratio for the hidden-layer
weights in a 2-layer MLP. Four different stages of learning are evaluated.
lations were performed based on the same choices of bit sizes for each component given in
Section 3.6 vs. different number of bits assigned to fw ij g and f1w ij g. Figure 9 shows the
average (of 256 training data) squared difference between the desired and actual outputs of
the 2-D regression problem after the network converges (a hard convergence is usually required
in this kind nonlinear regression problem). Note that, at the predicted point (around
15-16 bits of weights), the squared difference curve dives. That implies the inability to converge
to the desired mapping when the number of bits for the weights is less than 16 bits.
Similar supporting results are also observed in the XOR classification problem using an
MLP with 2 inputs, 3 hiddens, and 1 output (see Figure 10). Due to the classification nature
of the XOR problem, a soft convergence is good enough for the termination of training.
Therefore, at the predicted point of 12-13 bits of weights, the squared difference curve dives.
Another interesting observation is worthwhile to mention: the total finite precision error
in a single iteration of weight updating is mainly generated in the final jamming operators
in the computation of the output delta, hidden delta, and weight update. Therefore, even
though it is required to have at least 13 to 16 bits assigned to the computation of the weight
update and stored as the total weight value, the number of weight bits in the computation
of forward retrieving and hidden delta steps of learning can be as low as 8 bits without
excessive degradation of learning convergence and accuracy.0.020.060.10.140.184
Weight Bits
Average
Squared

Figure

9: The average squared differences between the desired and actual outputs of the
2-D regression problem after the network converges.
0.10.20.3
Weight Bits
Average
Squared

Figure

10: The average squared differences between the desired and actual outputs of the
problem after the network converges.
Concluding Remarks
The paper is devoted to the derivation of the finite precision error analysis techniques for
neural network implementations, especially analysis of the back-propagation learning of
MLP's. This analysis technique is proposed to be more versatile and to prepare the ground
for a wider variety of neural network algorithms: recurrent neural networks, competitive
learning networks, and etc. All these networks share similar computational mechanisms as
those used in back-propagation learning. For the forward retrieving operations, it is shown
that 8-bit weights are sufficient to maintain the same performance as using high precision
computation. On the other hand, for network learning, at least 14-16 bits of precision
must be used for the weights to avoid having the training process divert too much from the
trajectory of the high precision computation.



--R

A VLSI architecture for high-performance
Implementation limits for artificial neural networks.
From nonlinear optimization to neural network learning.
Parallel architectures for artificial neural nets.
A unified architecture for artificial neural networks.
Recursive least squares learning algorithms for neural net- works

Pizer with Victor L.
Learning internal representations by error propagation.
Beyond regression: New tools for prediction and analysis in the behavior science.
--TR
A unified systolic architecture for artificial neural networks
Learning internal representations by error propagation

--CTR
Ming Zhang , Stamatis Vassiliadis , Jos G. Delgado-Frias, Sigmoid Generators for Neural Computing Using Piecewise Approximations, IEEE Transactions on Computers, v.45 n.9, p.1045-1049, September 1996
Cesare Alippi , Luciano Briozzo, Accuracy vs. Precision in Digital VLSI Architectures for Signal Processing, IEEE Transactions on Computers, v.47 n.4, p.472-477, April 1998
Yongsoon Lee , Seok-Bum Ko, An FPGA-based face detector using neural network and a scalable floating point unit, Proceedings of the 5th WSEAS International Conference on Circuits, Systems, Electronics, Control & Signal Processing, p.315-320, November 01-03, 2006, Dallas, Texas
Cesare Alippi, Randomized Algorithms: A System-Level, Poly-Time Analysis of Robust Computation, IEEE Transactions on Computers, v.51 n.7, p.740-749, July 2002
feedforward neural nets;low precision;neural network algorithms;neural network hardware;multilayer perceptron;forward retrieving;silicon area;finite precision computation;neural chips;error analysis;back-propagation learning;parallel processing;system cost