Lightweight Neural Networks

arXiv:1712.05695v1 [cs.LG] 15 Dec 2017

Altaf H. Khan
altaf@altafkhan.com
336 Eden Canal Villas, Lahore, Pakistan
December 18, 2017
Abstract
Most of the weights in a Lightweight Neural Network have a value of zero, while the remaining
ones are either +1 or −1. These universal approximators require approximately 1.1 bits/weight of
storage, posses a quick forward pass and achieve classification accuracies similar to conventional
continuous-weight networks. Their training regimen focuses on error reduction initially, but later
emphasizes discretization of weights. They ignore insignificant inputs, remove unnecessary weights,
and drop unneeded hidden neurons. We have successfully tested them on the MNIST, credit card
fraud, and credit card defaults data sets using networks having 2 to 16 hidden layers and up to
4.4 million weights.

1

Lightweight Neural Networks

Lightweight Neural Networks (LWN) are a subset of the conventional Continuous-Weight Networks
(CWN). We call them lightweight because the trained LWNs have weights that require approximately
1.1 bits/weight of storage and their forward-passes does not require floating-point multiplications. The
key characteristic of LWNs is the sparsity of their weight matrices. Moreover, the non-zero weights of
these matrices are limited to only two values: ±1 (see Figure 1). These networks were first introduced
in 1996 [1, 2] as Multiplier-Free Networks and used training heuristics that were proposed in 1994 [3].
Due to the recent interest in similar networks [4, 5, 6, 7], we present new results highlighting the
sparsity of these networks, their natural inclination towards forming tight receptive fields, and their
universal approximation capability.

2

LWN and the Biological Neural Network

Here we would like to highlight those aspects of the LWN neurons that make them more similar in
structure and function to the biological neurons as compared with the CWN neurons. Consider an
axon of a source biological neuron connecting to the dendrite of the target neuron at a synapse. One of
two types of neurotransmitter chemicals (either for excitatory or inhibitory receptors) is released from
the axon’s side of the synapse whenever the source neuron is activated [8]. These chemicals then bind
with receptors on the dendrite-side of the synapse, resulting in an increase (in case of an excitatory
receptor) or decrease (inhibitory receptor) in the electrical potential on the membrane of the target
neuron. The electrical potential of that membrane is the sum of contributions due to the firings of all
neurons that are connected through synapses to the target neuron. When the membrane’s electrical
potential reaches a threshold value, the target neuron fires. The highlight of the above narrative is the
absence of multiplication operations and the presence of only two synaptic values, excitatory (similar
to the +1 weight of LWN connections) and inhibitory(−1 weight)1 .
The connections-to-neurons ratio decreases with increase in the number of neurons in biological
systems [9, 10]. LWNs exhibit the same characteristic (Figure 2), but CWNs can not. The size of the
1 LWN neurons are different from the biological neuron in having bipolar outputs as well as bipolar inputs. In case of
the negative-valued inputs, the roles of excitatory and inhibitory receptors are reversed.

1

x1

x2

x1

h11

h11

x2
h21

x3

h21

h12

x3

y1

h12

h22
x4

h13

y2

h13

x4

h23
x5

y2
h23

h14

x5

y3

h14

h24
x6

y1
h22

y3
h24

h15

x6

x7

h15

x7

Figure 1: Each CWN neuron is connected to all neurons in the preceding layer, whereas the LWN
neurons have limited receptive fields. CWN weights have varying values (depicted by the thickness of
the connections) and can be positive (solid lines) or negative (broken lines). The LWN weights have
just two values, ±1.
receptive field, NRF , of a biological neuron is the fan-in of that neuron. Studies of processing in the
visual cortex of animals show that NRF varies among different types of neurons [11]. The LWN is
similar in structure in that each LWN neuron is a specialist (as compared to the generalist neurons of
the CWN) as LWN neuron specializes in a particular subset of inputs. The NRF for a conventional
CWN is fixed for every neuron in every layer and is equal to the number of neurons in the preceding
layer. For LWN, NRF is much smaller and varies with the number of neurons in a layer.
LWN training is inspired by the synaptic pruning process in the biological brain [12]: start with
plenty; prune the excess off later. This natural phenomenon prunes, for example, 25-50% synapses
among humans as they approach adulthood, but does not reduce the number of neurons. For LWN,
the training process prunes the initial count of the synapses (i.e. weights) by about 80-95%, which, in
many cases, results in the elimination of some of the neurons as well.

3

Universal Approximation

Although their weights are restricted to the set {0, ±1}, LWNs’ thresholds have no such constraints.
Their activation functions are bounded and odd, and are confined to hyperbolic tangent in this discussion. With the help of these ingredients, we can create a one-dimensional bump of arbitrarily small
height at an arbitrary location on the x-axis using expressions of the form
tanh(x − ρ) + tanh(−(x − ρ) + χ),
where ρ, χ ∈ R. Summations of such bumps can be used to approximate arbitrary one-dimensional
functions to any accuracy. We will now discuss the extension of this one-dimensional construction to
denseness in C(Rn ). We start by a theorem due to Khan [13]:
Theorem 1 (Weightless Neural Network Existence Theorem) Summations of the form:
!
!
K
I
L
I
X
X
X
X
σ
xi − ck +
ψ
x i − cl
k

i

l

n

i

are dense in C(R ), where ck , cl ∈ R, σ(·) : R → R is bounded and odd, ψ(.) = ασ(.), and α ∈ R\Q.
2

Figure 2: Ratio of number of non-zero weights to number of functioning neurons, N±1 /M f unctioning ,
against number of functioning neurons in LWNs on the MNIST data set

100
N±1 /M functioning = -5.642log 2 M functioning + 109.069

90

R2 = 0.625

N±1 /M functioning

80
70
60
50
40
26

28

27

Number of Functioning Neurons,

29

2 10

M functioning = M total − M dropped

The Weightless Neural Network is, on one hand, simpler than CWN in having unit-valued input- and
output-layer weights, but on the other hand more complex in having two types of hidden neurons. The
two types differing only by an irrational multiplication factor, α, in their activation functions.
Allowing the input- and output-layer weights to assume additional values of 0 and −1 does not
harm the density result, but may result in networks that train quicker and are more compact. The
choice of multiplication factor α is arbitrary, and can be chosen to be close to one. When simulated on a
digital computer, that choice will become exactly one due to the limited precision of the computer [14],
and the network will end up having only a single type of hidden neurons. This simplified configuration
having I inputs and H hidden neurons can be written as:
!
H
I
X
X
ah σ
bih xi − ch
k

i

where ah , bhi ∈ {0, ±1}, ch ∈ R. This expression depicts a network having a solitary layer of hidden
neurons. To form multi-hidden-layer networks, layers comprising neurons identical to those in the
first hidden-layer were employed. Moreover, for our simulations, we used output neurons which were
identical to the hidden neurons.

4

Training Procedure

Several approaches have been proposed for training neural networks with discrete weights. Hwang
and Sung [4] take a trained CWN, discretize its weights to ternary {0, ±1} values, and retrain using
backpropagation. They also restricted all signals to a depth of three bits. Mellempudi et al. [7] also
start with a trained CWN and use a fine-grained ternarization method which exploits local correlations
in the dynamic-range of the parameters to minimize the impact of discretization on the accuracy of the
network. Li et al. [5] use the stochastic gradient descent method to train ternary-weight networks. They
use ternary-weights during the forward and backward propagations, but not during the weight updates.
Yin et al. [6] gradually discretize the weights to zero or powers of two by minimizing the Euclidean
distance between conventional weights and their closest discrete value during backpropagation.
We use the training heuristics proposed by Khan [3] augmented by a new step with the aim of
minimizing the error:
3

Figure 3: Plot of training epochs against mean-squared error on training data, ratio of non-discrete
weights to all weights, and test data miss-classifications. The bottom plot is the same as the top one,
but with a magnified y-axis

1.0
0.8
0.6
0.4
0.2
0.0
0.05
0.04
0.03
0.02
0.01
0.00

Test Data Miss-classifications
Non-Discrete Weights
Training MSE
0

100

200

E = Eo + Ew =

300
400
500
Training Epochs

X

|o − t|2 +

all examples

600

X

700

800

|w − Q(w)|2

all weights

where o and t are the calculated and desired output vectors, w the value of an individual weight, and
Q(·) the weight discretization function. Q(·) is differentiable and the zeros of (w − Q(w)) are {0, ±1}.
The main point of the original heuristics was the sequential application of error-reduction and weightdiscretization steps to the network in every training epoch. Both were based on steepest-descent, but
weight-discretization was supplemented by an additional mechanism to take care of the weight-update
paralysis. This paralysis was caused by the opposing weight-updates calculated by the error-reduction
and weight-discretization steps. That additional mechanism - the black-hole mechanism - forced nearlydiscrete weights to discrete values. The rate of weight-discretization and the radius of the black-hole
grew as the error in the output of the network shrank.
The black-hole mechanism worked well for shallow networks comprising hundreds of weights as
discussed in [3], but failed to overcome the weight-update paralysis for deeper LWNs having thousands
or millions of weights discussed in this paper. For such networks, we propose an additional mechanism
that comes into play only when almost all weights are discrete. At that stage, all weights are rounded
to their nearest discrete value from the set {0, ±1}. If that results in a network having acceptable
test-data accuracy, training is concluded. Otherwise, the rounding step is rolled back and normal
training is resumed with the pre-rounding weights. The resulting error plots for a typical LWN are
shown in Figure 3.

5

MNIST Data Set Simulations

MNIST is a well-known data set of images of handwritten digits [15, 16]. We used the version available
through the TensorFlow machine learning library [17]. That version includes 55,000 images in the
training set and 10,000 in the test set. All images have been normalized, centered and transformed to
a 28 × 28 matrix of 32-bit floating-point numbers ranging from zero to one according to the gray-scale
value of the associated pixel. The 28 × 28 matrices have been flattened to 784-element vectors. Image
labels are 10-element vectors with only a single element in a vector equal to one, and the rest having
a value of zero. We used this data set as is and did not try to help the network configuration or
the training process with any information about the original, 2-D nature of the image vector. This is
4

Figure 4: Relationship of the non-zero weights, N±1 , and the number of neurons in the hidden layers,
H1 , H2 , on the MNIST data set

2 17
2 16

N±1

2 15
2 14
2 13
2 12
2 11
10

log2 N± =
1 0
23

24

.689log

25

H1

2 H1 +
2

0.10
28
7
R = 0.93log2 H2 + 8.093
2
85
26
H2
26
5
27
28

29 24

29

2

2

because our interest is not to get the best possible result but to look at what the LWN learns with the
simplest possible configuration. In this simplest configuration, all neurons in all layers are exactly the
same and all layers are fully-connected to layer preceding them. The only factors that we varied were
the number of hidden-layers and the number of neurons in each of those layers.
The summary of our results for two-hidden-layer LWNs is shown in Table 1. For each configuration,
the results for the top-two LWNs with the best test-data accuracies are shown. The test-data accuracies
range from 93.8 to 97.0%. If we want to balance accuracy with size, we will choose the 784:256:128:10
LWN. It achieves an accuracy of 96.7% with 0.23 million weights which compares favorably with
97.0% for a much larger LWN having 1,152 extra neurons and 1.1 million additional weights. This
result (Table 2) is similar to 96.95% reported for a 784:300:100:10 CWN and 97.05% reported for a
784:500:150:10 CWN [16]. We do not have much to say about the epochs required to achieve reasonable
accuracy as that number is very much dependent on the training parameters. We did not make any
effort to optimize the parameters for that purpose.
Figure 3 shows the mean-squared error on training data, the ratio of misclassifications to the
total number of test data examples, and the ratio of non-discrete weights to the total number of
weights as a function of training epochs. That figure clearly reflects the very deliberate slowness of the
weight-discretization processes as compared with the error minimization process. Our intention was
to find the continuous-weight error minimum first and then look for the discrete-weight minimum in
its immediate vicinity. The lack of smoothness in the training error and test misclassification curves is
due to the weight-discretization corrections. The smoothness of the non-discrete weight curve indicates
the slowness of discretization.

5.1

Zero-Valued Weights

The sparsity, N0 , of the two-hidden-layer LWNs ranged from 79.5 to 96%, with the larger networks
tending to have sparser weight matrices. The relationship of the dimensions of the hidden layers, H1
and H2 , and the number of non-zero weights, N±1 , is shown in Figure 4 on a log-log-log scale. N±1
seems to be mainly dependent on H1 .
5.1.1

Ignored Inputs

The all-zero-rows column in Table 1 indicates that the LWN ignores some of the inputs when the
number of neurons in the first hidden layer is inadequate. For the MNIST data set, the minimum layer
size seems to be 256. This number is consistent with a minimum of 300 that was reported by LeCun et
al. [18]. For the 784:32:16:10 LWNs, the ignored pixels were in the top five and bottom three rows, and
four left-most and three right-most columns. Whereas for the larger 784:128:64:10 LWNs the ignored

5

Table 1: Two-hidden-layer LWN results for the MNIST data

Configuration

784:30:14:10
784:32:16:10
784:64:16:10
784:64:32:10
784:128:16:10
784:128:32:10
784:128:64:10
784:256:16:10
784:256:32:10
784:256:64:10
784:256:128:10
784:512:16:10
784:512:32:10
784:512:64:10
784:512:128:10
784:512:256:10
784:1024:16:10
784:1024:32:10
784:1024:64:10
784:1024:128:10
784:1024:256:10
784:1024:512:10

Epochs
2,005
3,670
2,958
6,625
1,713
1,129
2,324
615
500
407
739
1,356
698
472
403
683
452
672
493
840
797
488
540
432
425
547
579
549
707
856
742
783
390
603
480
619
507
765
799
727
1,129
859
1,520
1,532

Test
Number of All-Zero Rows in
Maximum
Data
the Weight Matrix of the
Fan-In
N
N±1
N0
Accuracy
1st Hidden 2nd Hidden Output of Output
(%)
(%)
Layer
Layer
Layer
Neurons
93.8
3,756 84.4
124
0
0
3
24,080
93.8
3,953 83.6
124
0
0
2
94.6
4,520 82.5
111
0
4
4
25,760
94.8
5,281 79.5
118
0
3
3
95.4
6,322 87.7
66
0
5
2
51,360
95.1
6,276 87.8
60
0
1
3
95.5
6,216 88.2
65
1
10
2
52,544
95.2
5,762 89.0
53
0
12
2
95.4
10,620 89.6
27
0
1
3
102,560
95.3
10,931 89.3
17
1
4
2
95.7
10,415 90.1
20
0
21
2
104,768
95.6
10,883 89.6
30
0
16
1
95.4
10,032 90.8
19
0
45
2
109,184
95.6
10,326 90.5
16
0
45
2
96.0
18,625 90.9
1
5
2
2
204,960
95.9
17,512 91.5
0
6
3
4
96.1
17,715 91.5
4
0
15
1
209,216
96.1
18,317 91.2
3
2
17
2
96.5
18,336 91.6
3
0
44
3
217,728
96.4
18,619 91.4
1
0
42
2
96.7
16,996 92.8
2
1
109
2
234,752
96.3
17,475 92.6
3
0
99
3
96.4
29,867 92.7
0
13
3
2
409,760
96.2
28,143 93.1
0
32
5
2
96.3
29,474 93.0
0
16
17
2
418,112
96.1
29,701 92.9
0
15
14
2
96.2
30,012 93.1
0
2
48
2
434,816
96.0
31,954 92.7
0
6
44
2
96.5
31,288 93.3
0
0
80
2
468,224
96.3
33,067 92.9
0
2
102
2
96.7
32,556 93.9
0
0
207
2
535,040
96.1
34,556 93.5
0
0
213
2
96.2
42,283 94.8
0
244
2
2
819,360
96.6
32,934 96.0
0
258
4
2
96.3
46,399 94.4
0
96
18
2
835,904
96.2
43,574 94.8
0
96
18
2
96.6
48,066 94.5
0
37
48
1
868,992
96.4
50,851 94.1
0
15
48
2
96.1
57,133 93.9
0
2
97
3
935,168
96.3
54,955 94.1
0
2
96
2
96.3
60,366 94.3
0
0
213
2
1,067,520
96.0
62,125 94.2
0
0
201
3
96.2
73,255 94.5
8
0
419
2
1,332,224
97.0
76,459 94.3
10
0
446
2

6

Table 2: Best performing LWNs and CWNs on the MNIST data set. The boldface numbers highlight
changes from the starting configurations.
Network
CWN [18]
CWN [18]
LWN
LWN

Test Data
Accuracy (%)
96.95
97.05
96.68
97.02

Configuration When
Training Started
784:300:100:10
784:500:150:10
784:265:128:10
784:1024:512:10

N
266,200
468,500
234,752
1,332,224

Configuration at
the End of Training
n.a.
n.a.
780:262:28:10
774:1024:66:10

N±1
n.a.
n.a.
16,996
76,459

pixels were in the top five and bottom row, and three left-most and two right-most columns. This
indicates that the normalized MNIST images are slightly off-center and the LWNs are ignoring the
nearly-white pixels around the digits due to their low information value.
5.1.2

Receptive Fields

Due to its fully-connected structure, the weights-to-neurons ratio for CWN increases with the number
of neurons. The LWN restricts this rise in complexity by limiting each neuron to a narrow receptive
field. Figure 2 shows the weights-to-neurons ratio as a function of the number of neurons in the
network.
average
, as
Variations in the average size of the receptive field of neurons in the first hidden-layer, NRF
a function of the size of the hidden layer is shown in Figure 5. Here, we excluded the smaller networks,
those with 30-32 neurons in the hidden-layer, from the linear-regression model. The LWN reduces the
burden of processing on each neuron by reducing their NRF as more and more of them are added to
the first hidden-layer.
We emphasize here that the LWN has no way of knowing that the MNIST data set comprises
2-D images. Neither does the LWN architecture include any custom-designed, small-receptive-field
convolution layers of the Convolution Neural Network (CNN) à la LeCun et al. [16]. It, however, tends
to drop non-crucial inputs from the receptive-field of neurons by setting the corresponding weight to
zero. This behavior is true for almost all neurons in all layers of the network. In contrast with the
CNN, the size of the receptive field varies from neuron to neuron in an LWN and is not a parameter
of the training process, but is arrived at naturally as a consequence of the training process. Receptivefields are clearly present in biological systems for the processing of visual, aural and touch and possibly
other stimuli. The design of CNN, for example, is inspired by those systems. There may be other dataprocessing situations where the receptive-fields may not be that obvious. The LWN can automatically
discover and leverage them even in those situations.
5.1.3

Dropped Hidden Neurons

Some of the hidden neurons had all their weights set to zero as a consequence of the training regimen.
The number of such dropped neurons, indicated by the number of all-zero rows in the weight matrices,
is related to the mismatch between the number of neurons in consecutive layers. Table 1 makes this
clear: greater the mismatch, the greater the number of dropped hidden neurons.
Tables 1 also shows that, in 70% of the cases, a hidden-layer neuron is connected to at most two
output neurons. This indicates the efficient distribution of the image recognition task among the
hidden neurons. That is, the task is distributed in such a way that each neuron has a distinct and
focused responsibility. Here the size of the hidden-layer does not seem to be much of a factor. In the
most extreme case, there are 446 dropped neurons in the second hidden layer. Of the remaining 66,
61 are connected only to single output neuron, while the other five are connected to two of the output
average
neurons. The NRF
for the output neurons was 7.1.
5.1.4

Weight Storage

Each LWN weight requires at most two binary-bits for storage. However, using variable-length coding
(e. g. Huffman coding [19]), a weight matrix that is 90% sparse will require only 1.1 bits/weight. This

7

average
Figure 5: Relationship of the average size of the receptive fields, NRF
, and the number of neurons
in the first hidden-layer, H1 , on the MNIST data set. The regression model excludes the first four
points.

160
140

N±average
= -10.880log 2 H1 + 151.777
1

R2 = 0.936

average
NRF

120
100
80
60
40
20

25

26

28

27

29

2 10

H1
Table 3: Weight storage comparison for the MNIST data set for the 784:1024:512:10 configuration
that has N = 1, 332, 224 weights
N±1 /N

Storage Scheme

Bits/Weight

Required Storage (kB)

n.a.
n.a.
0.943
0.943

Conventional
Conventional
Conventional
Huffman coding

32
8
2
1.057

5,204
1,301
325
172

number approaches one as sparsity increases. Table 3 shows a comparison of storage requirements for
various weight depths, but excludes the effect of thresholds on storage requirements. 32- and 8-bit
thresholds for the 784:1024:512:10 configuration consume 48 and 12 kB, respectively. We have yet
to try LWN training with 8-bit thresholds, but we expect it to work due to the results reported by
Vanhoucke et al. [20].
5.1.5

Forward Pass

LWNs posses an efficient forward pass because multiplications in all neurons are trivial in nature
as weights are restricted to the set {0, ±1}. Moreover, due to the high sparsity of weight matrices,
even that trivial operation is not necessary most of the time. The thresholds are 32-bit floating-point
numbers, but they are never part of a multiplication operation. Therefore, the LWN does not require
the floating-point multiplication operation at all.

5.2

Training Deeper Networks

We trained several LWNs with 4-16 hidden-layers of varying sizes to see if our training heuristics
suffered from the vanishing (and/or exploding) gradient problem [21, 22]. Results are shown in Table 4.
We did these training runs using the same training parameter values (e.g. learning rate, momentum,
discretization rates) as those of the 2-hidden-layer LWNs. Our focus was not on obtaining the best
performing LWNs, but to see if we were able to train theses deeper LWNs at all. Surprisingly, these

8

Table 4: MNIST training results for LWNs having 4-16 hidden layers
N0
Test Data
Hidden
N
N±1
Epochs
(%)
Accuracy (%)
Layers
339
96.3
116,244 92.2
784:1024:512:128:64:10
4
1,499,776
469
97.0
98,018 93.5
481
96.8
19,364 92.1
784:256:128:64:32:16:10
5
244,384
874
96.9
19,506 92.0
321
96.3
46,192 92.0
784:512:256:128:64:32:10
5
575,808
330
96.0
48,010 91.7
630
96.0
255,229 94.2
784:2048:1024:512:256:128:64:32:16:10
8
4,401,824
1,081
95.8
264,572 94.0
771
96.5
21,664 91.7
784:256:128:{5×64}:32:16:10
9
260,768
655
96.6
21,198 91.9
822
95.7
25,732 91.1
784:256:128:{12×64}:32:16:10
16
289,440
862
95.7
26,280 90.9
Configuration

Table 5: Test data results for the credit card fraud data.
Sampling
Method
Undersampled
Oversampled
SMOTE

Configuration

N

N±1

29:256:128:64:32:2

50,496

1,710
6,389
7,655

N0
(%)
96.6
87.3
84.8

Accuracy
(%)
97.7
99.9
99.9

Precision

Recall

0.058
0.857
0.829

0.804
0.811
0.818

F1
Score
0.109
0.833
0.823

training experiments did not present symptoms of the vanishing or exploding gradient. This is probably
due to the restriction on the magnitude of the weights.

6

Credit Card Fraud and Defaults Simulations

After extensive simulations on the MNIST data, we looked at credit card fraud [23] and default [24]
data to further validate the viability of the LWN. A key characteristic of these data sets is their class
imbalance. This issue can be addressed by undersampling the majority class or oversampling the
minority class. We tried oversampling in two different ways: simple repetition of the minority class
examples and Synthetic Minority Oversampling Technique (SMOTE) [25]. For these simulations, we
normalized the continuous features to zero-mean and unit-variance and clipped them to the range
[−1, 1]. All binary features were mapped to {−1, 1}. We then split the data into training (70%) and
testing (30%) sets using stratified sampling. We assigned separate outputs to each of the two classes.
The higher of the two outputs was considered the winning class while testing. This methodology avoids
the work required to find the most suitable output threshold (in case of a single output neuron) at
which the classification switches from one class to the other.
The credit card fraud data set contained 29 features and consisted of 284,807 transactions, out
of which 0.17% were fraudulent. Our best performing network had an F1 score [26] of 0.83 with
oversampled training data. Pozzolo et al. have reported G-mean scores of 0.944, 0.770, and 0.794
on this data set using Logit Boost, Random Forests, and Support Vector Machine, respectively [23].
It is difficult to compare these results with our G-mean score of 0.900 due to, among other things,
differences in data pre-processing, training/testing data splits, and classification thresholds.
The credit card defaults data set contained 23 features and consisted of 30,000 cases, out of which
22% were defaults. We replaced categorical features with separate features for each category. The three
sampling techniques, in this case, resulted in very similar F1 scores (Table 6) due to the relatively lower
imbalance among the two classes. Yeh et al. have reported 83% accuracy using the neural network
classifier on this data set. It is not possible to compare this result with our result of 68.5% due to lack
of adequate information about their training and testing processes.

9

Table 6: Test data results for the credit card default data.
Sampling
Configuration
N
N±1
Method
Undersampled
22,528
Oversampled 32:512:256:128:64:32:16:8:4:2 191,144 37,648
SMOTE
46,449

7

F1
N0 Accuracy
Precision Recall
Score
(%)
(%)
88.2
68.5
0.365
0.571 0.445
80.3
68.4
0.368
0.596 0.455
75.7
67.5
0.359
0.595 0.448

Conclusions

LWNs do not have weights in the conventional sense, just excitatory and inhibitory connections of
unit strength. They can approximate any continuous function with any accuracy. They require modest storage and have a multiplication-free forward pass, rendering them suitable for deployment on
inexpensive hardware. Their sparse weight matrices loosen the coupling among the layers, making the
LWN more tolerant to failure of individual neurons. In a CWN, the learned information is distributed
over all weights. In an LWN, the picture is less fuzzy and the localized nature of computation is much
more obvious due to the presence of a large number of zero-valued weights. For image processing, a
CWN does not scale well with increases in the resolution of the images due to CWN’s fully-connected
structure. An LWN, on the other hand, scales much better due to the sparsity of its weight matrices.
The small magnitude of LWN weights should result in smooth mappings [27] and the small number of
non-zero weights should result in low generalization error [28].
The LWN learning process automatically drops insignificant inputs, unnecessary weights, and unneeded hidden-neurons. That process is relatively complex and slow, but results in networks that are
almost as accurate as the CWNs but have much lower information complexity. It is conjectured that
those accuracies can be matched by using bigger networks. This can be understood by considering
the limited number of angles at which an LWN neuron can draw classification boundaries as opposed
to the CWN neuron that can draw those boundaries at an arbitrary angle. A superposition of two or
more LWN neurons can, however, approximate those arbitrary boundaries, but the added complexity
may not justify the minuscule improvement in approximation accuracy.
At this time, the LWN neurons have unrestricted thresholds. We are exploring if the magnitude of
these thresholds can be restricted to an arbitrary value [29], e.g. 1, which may lead to more efficient
storage. In its current form, the LWN training process applies the same training operation to every
weight irrespective of the value of the weight. We are trying to see if the training process can be made
more efficient by sometimes ignoring those weights that have not deviated from a zero value for several
epochs.

References
[1] A. H. Khan, Feedforward neural networks with constrained weights. PhD thesis, University of
Warwick, 1996.
[2] A. H. Khan, “Multiplier-free feedforward networks,” in International Joint Conference on Neural
Networks (IJCNN), vol. 3, pp. 2698–2703, IEEE, 2002.
[3] A. H. Khan and E. L. Hines, “Integer-weight neural nets,” Electronics Letters, vol. 30, no. 15,
pp. 1237–1238, 1994.
[4] K. Hwang and W. Sung, “Fixed-point feedforward deep neural network design using weights +1,
0, and −1,” in IEEE Workshop on Signal Processing Systems (SiPS), pp. 1–6, IEEE, 2014.
[5] F. Li, B. Zhang, and B. Liu, “Ternary weight networks,” in The 1st International Workshop on
EMDNN, NIPS, 2016.
[6] P. Yin, S. Zhang, J. Xin, and Y. Qi, “Quantization and training of low bit-width convolutional
neural networks for object detection,” ArXiv Preprint arXiv:1612.06052v2, 2017.

10

[7] N. Mellempudi, A. Kundu, D. Mudigere, D. Das, B. Kaul, and P. Dubey, “Ternary neural networks
with fine-grained quantization,” ArXiv Preprint arXiv:1705.01462, 2017.
[8] M. F. Bear, B. W. Connors, and M. A. Paradiso, Neuroscience, vol. 2. Lippincott Williams &
Wilkins, 2007.
[9] J. L. Ringo, “Neuronal interconnection as a function of brain size,” Brain, Behavior and Evolution,
vol. 38, no. 1, pp. 1–6, 1991.
[10] D. K. Cullen, M. E. Gilroy, H. R. Irons, and M. C. LaPlaca, “Synapse-to-neuron ratio is inversely
related to neuronal density in mature neuronal cultures,” Brain Research, vol. 1359, pp. 44–55,
2010.
[11] D. H. Hubel and T. N. Wiesel, “Receptive fields and functional architecture of monkey striate
cortex,” The Journal of Physiology, vol. 195, no. 1, pp. 215–243, 1968.
[12] G. Chechik, I. Meilijson, and E. Ruppin, “Synaptic pruning in development: A computational
account,” Neural Computation, vol. 10, no. 7, pp. 1759–1777, 1998.
[13] A. H. Khan, “Weightless neural net is a universal approximator,” 2017. Manuscript submitted for
publication.
[14] J. Wray and G. G. Green, “Neural networks, approximation theory, and finite precision computation,” Neural Networks, vol. 8, no. 1, pp. 31–37, 1995.
[15] Y. LeCun, C. Cortes, and C. J. Burges, “MNIST handwritten digit database,” AT&T Labs [Online], vol. 2, 2010. The data set is available at http://yann.lecun.com/exdb/mnist.
[16] Y. LeCun, B. E. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. E. Hubbard, and L. D.
Jackel, “Handwritten digit recognition with a back-propagation network,” in Advances in Neural
Information Processing Systems, pp. 396–404, 1990.
[17] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis,
J. Dean, M. Devin, et al., “Tensorflow: Large-scale machine learning on heterogeneous distributed
systems,” ArXiv Preprint arXiv:1603.04467, 2016.
[18] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document
recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
[19] D. A. Huffman, “A method for the construction of minimum-redundancy codes,” Proceedings of
the IRE, vol. 40, no. 9, pp. 1098–1101, 1952.
[20] V. Vanhoucke, A. Senior, and M. Z. Mao, “Improving the speed of neural networks on CPUs,” in
Proceedings of Deep Learning and Unsupervised Feature Learning NIPS Workshop, vol. 1, p. 4,
2011.
[21] S. Hochreiter, “Untersuchungen zu dynamischen neuronalen netzen,” Diploma, Technische Universität München, vol. 91, 1991.
[22] J. Schmidhuber, “Deep learning in neural networks: An overview,” Neural Networks, vol. 61,
pp. 85–117, 2015.
[23] A. Dal Pozzolo, O. Caelen, R. A. Johnson, and G. Bontempi, “Calibrating probability with undersampling for unbalanced classification,” in IEEE Symposium Series on
Computational Intelligence, pp. 159–166, IEEE, 2015.
The data set is available as
https://www.kaggle.com/dalpozz/creditcardfraud/downloads/creditcardfraud.zip.
[24] I.-C. Yeh and C.-h. Lien, “The comparisons of data mining techniques for the predictive accuracy
of probability of default of credit card clients,” Expert Systems with Applications, vol. 36, no. 2,
pp. 2473–2480, 2009. The data set is available as https://archive.ics.uci.edu/ml/machine-learningdatabases/00350/default of credit card clients.xls.
11

[25] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “SMOTE: Synthetic minority
over-sampling technique,” Journal of Artificial Intelligence Research, vol. 16, pp. 321–357, 2002.
[26] T. Fawcett, “An introduction to ROC analysis,” Pattern Recognition Letters, vol. 27, no. 8,
pp. 861–874, 2006.
[27] C. M. Bishop, Neural Networks for Pattern Recognition. Oxford University Press, 1995.
[28] J. E. Moody, “The effective number of parameters: An analysis of generalization and regularization
in nonlinear learning systems,” in Advances in Neural Information Processing Systems, pp. 847–
854, 1992.
[29] M. Stinchcombe and H. White, “Approximating and learning unknown mappings using multilayer feedforward networks with bounded weights,” in International Joint Conference on Neural
Networks (IJCNN), pp. 7–16, IEEE, 1990.

12

