abstract: We introduce a method to train Quantized Neural Networks (QNNs) — neural networks with extremely low precision (e.g., 1-bit) weights and activations, at run-time. At train-time the quantized weights and activations are used for computing the parameter gradients. During the forward pass, QNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations. As a result, power consumption is expected to be drastically reduced. We trained QNNs over the MNIST, CIFAR-10, SVHN and ImageNet datasets. The resulting QNNs achieve prediction accuracy comparable to their 32-bit counterparts. For example, our quantized version of AlexNet with 1-bit weights and 2-bit activations achieves $51\%$ top-1 accuracy. Moreover, we quantize the parameter gradients to 6-bits as well which enables gradients computation using only bit-wise operation. Quantized recurrent neural networks were tested over the Penn Treebank dataset, and achieved comparable accuracy as their 32-bit counterparts using only 4-bits. Last but not least, we programmed a binary matrix multiplication GPU kernel with which it is possible to run our MNIST QNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The QNN code is available online.
author: Itay Hubara\* firstname.lastname@example.com
Department of Electrical Engineering
Technion - Israel Institute of Technology
Haifa, Israel Matthieu Courbariaux\* firstname.lastname@example.com
Department of Computer Science and Department of Statistics
Université de Montréal
Montréal, Canada Daniel Soudry firstname.lastname@example.com
Department of Statistics
Columbia University
New York, USA Ran El-Yaniv email@example.com
Department of Computer Science
Technion - Israel Institute of Technology
Haifa, Israel Yoshua Bengio firstname.lastname@example.com
Department of Computer Science and Department of Statistics
Université de Montréal
Montréal, Canada

\*Indicates first authors.
bibliography: sample.bib
title: Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations

# Introduction

Deep Neural Networks (DNNs) have substantially pushed Artificial Intelligence (AI) limits in a wide range of tasks, including but not limited to object recognition from images , speech recognition , statistical machine translation , Atari and Go games , and even computer generation of abstract art .

Training or even just using neural network (NN) algorithms on conventional general-purpose digital hardware (Von Neumann architecture) has been found highly inefficient due to the massive amount of multiply-accumulate operations (MACs) required to compute the weighted sums of the neurons' inputs. Today, DNNs are almost exclusively trained on one or many very fast and power-hungry Graphic Processing Units (GPUs) . As a result, it is often a challenge to run DNNs on target low-power devices, and substantial research efforts are invested in speeding up DNNs at run-time on both general-purpose and specialized computer hardware .

The most common approach is to compress a trained (full precision) network. HashedNets reduce model sizes by using a hash function to randomly group connection weights and force them to share a single parameter value. compressed deep convnets using vector quantization, which resulteds in only a $1\%$ accuracy loss. However, both methods focused only on the fully connected layers. A recent work by successfully pruned several state-of-the-art large scale networks and showed that the number of parameters could be reduced by an order of magnitude.

Recent works have shown that more computationally efficient DNNs can be constructed by quantizing some of the parameters during the training phase. In most cases, DNNs are trained by minimizing some error function using Back-Propagation (BP) or related gradient descent methods. However, such an approach cannot be directly applied if the weights are restricted to binary values. used a variational Bayesian approach with Mean-Field and Central Limit approximation to calculate the posterior distribution of the weights (the probability of each weight to be +1 or -1). During the inference stage (test phase), their method samples from this distribution one binary network and used it to predict the targets of the test set (More than one binary network can also be used). similarly used two sets of weights, real-valued and binary. They, however, updated the real valued version of the weights by using gradients computed by applying forward and backward propagation with the set of binary weights (which was obtained by quantizing the real-value weights to +1 and -1).

This study proposes a more advanced technique, referred to as Quantized Neural Network (QNN), for quantizing the neurons and weights during inference and training. In such networks, all MAC operations can be replaced with $XNOR$ and $population\, count$ (i.e., counting the number of ones in the binary number) operations. This is especially useful in QNNs with the extremely low precision — for example, when only 1-bit is used per weight and activation, leading to a Binarized Neural Network (BNN). The proposed method is particularly beneficial for implementing large convolutional networks whose neuron-to-weight ratio is very large.
This paper makes the following contributions:

-   We introduce a method to train Quantized-Neural-Networks (QNNs), neural networks with low precision weights and activations, at run-time, and when computing the parameter gradients at train-time. In the extreme case QNNs use only 1-bit per weight and activation(i.e., Binarized NN; see Section ).

-   We conduct two sets of experiments, each implemented on a different framework, namely Torch7 and Theano, which show that it is possible to train BNNs on MNIST, CIFAR-10 and SVHN and achieve near state-of-the-art results (see Section ). Moreover, we report results on the challenging ImageNet dataset using binary weights/activations as well as quantized version of it (more than 1-bit).

-   We present preliminary results on quantized gradients and show that it is possible to use only 6-bits with only small accuracy degradation.

-   We present results for the Penn Treebank dataset using language models (vanilla RNNs and LSTMs) and show that with 4-bit weights and activations Recurrent QNNs achieve similar accuracies as their 32-bit floating point counterparts.

-   We show that during the forward pass (both at run-time and train-time), QNNs drastically reduce memory consumption (size and number of accesses), and replace most arithmetic operations with bit-wise operations. A substantial increase in power efficiency is expected as a result (see Section ). Moreover, a binarized CNN can lead to binary convolution kernel repetitions; we argue that dedicated hardware could reduce the time complexity by $60\%$ .

-   Last but not least, we programmed a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy (see Section ).

-   The code for training and applying our BNNs is available on-line (both the Theano [^1] and the Torch framework [^2]).

# Binarized Neural Networks

In this section, we detail our binarization function, show how we use it to compute the parameter gradients, and how we backpropagate through it.

## Deterministic vs Stochastic Binarization

When training a BNN, we constrain both the weights and the activations to either $+1$ or $-1$. Those two values are very advantageous from a hardware perspective, as we explain in Section . In order to transform the real-valued variables into those two values, we use two different binarization functions, as proposed by . The first binarization function is deterministic: $$x^b = {\rm sign}(x) = \left\{ \begin{array}{ll}
                        +1 & \mbox{if $x \geq 0$},\\
                        -1 & \mbox{otherwise},\end{array} \right.$$ where $x^b$ is the binarized variable (weight or activation) and $x$ the real-valued variable. It is very straightforward to implement and works quite well in practice. The second binarization function is stochastic: $$\label{eq:sampled-wb}
    x^b = \left\{ \begin{array}{ll}
            +1 & \mbox{with probability $p = \sigma(x)$},\\
            -1 & \mbox{with probability $1-p$},\end{array} \right.$$ where $\sigma$ is the *"hard sigmoid"* function: $$\sigma(x) = {\rm clip}(\frac{x+1}{2},0,1) = \max(0,\min(1,\frac{x+1}{2})).$$ This stochastic binarization is more appealing theoretically (see Section ) than the sign function, but somewhat harder to implement as it requires the hardware to generate random bits when quantizing . As a result, we mostly use the deterministic binarization function (i.e., the sign function), with the exception of *activations at train-time* in some of our experiments.

## Gradient Computation and Accumulation

Although our BNN training method utilizes binary weights and activations to compute the parameter gradients, the real-valued gradients of the weights are accumulated in real-valued variables, as per Algorithm . Real-valued weights are likely required for Stochasic Gradient Descent (SGD) to work at all. SGD explores the space of parameters in small and noisy steps, and that noise is *averaged out* by the stochastic gradient contributions accumulated in each weight. Therefore, it is important to maintain sufficient resolution for these accumulators, which at first glance suggests that high precision is absolutely required.

Moreover, adding noise to weights and activations when *computing* the parameter gradients provide a form of regularization that can help to generalize better, as previously shown with variational weight noise , Dropout  and DropConnect . Our method of training BNNs can be seen as a variant of Dropout, in which instead of randomly setting half of the activations to zero when computing the parameter gradients, we binarize both the activations and the weights.

## Propagating Gradients Through Discretization

The derivative of the sign function is zero almost everywhere, making it apparently incompatible with back-propagation, since the exact gradients of the cost with respect to the quantities before the discretization (pre-activations or weights) are zero. Note that this limitation remains even if stochastic quantization is used. studied the question of estimating or propagating gradients through stochastic discrete neurons. He found that the fastest training was obtained when using the "straight-through estimator," previously introduced in Hinton's lectures . We follow a similar approach but use the version of the straight-through estimator that takes into account the saturation effect, and does use deterministic rather than stochastic sampling of the bit. Consider the sign function quantization $$q = {\rm Sign}(r),$$ and assume that an estimator $g_q$ of the gradient $\frac{\partial C}{\partial q}$ has been obtained (with the straight-through estimator when needed). Then, our straight-through estimator of $\frac{\partial C}{\partial r}$ is simply $$\label{eq:straight-through-gradient}
g_r = g_q 1_{|r|\leq 1}.$$ Note that this preserves the gradient information and cancels the gradient when $r$ is too large. Not cancelling the gradient when $r$ is too large significantly worsens performance. To better understand why the straight-through estimator works well, consider the stochastic binarization scheme in Eq. () and rewrite $\sigma(r)=\left(\mathrm{HT}(r)+1\right)/2$, where $\mathrm{HT}(r)$ is the well-known "hard tanh", $$\mathrm{HT}(r) = \left\{ \begin{array}{ll}
            +1 & \mbox{$r>1$},\\
             r & \mbox{$r\in[-1,1]$},\\
            -1 & \mbox{$r<-1$}.\end{array} \right. \label{eq:hard Tanh}$$ In this case the input to the next layer has the following form, $$\mathbf{W}_{b}h_{b}\left(\mathbf{r}\right)=\mathbf{W}_{b}\mathrm{HT}\left(\mathbf{r}\right)+\mathbf{n}\left(\mathbf{r}\right)\,,$$ where we use the fact that $\mathrm{HT}\left(\mathbf{r}\right)$ is the expectation over $h_{b}\left(\mathbf{x}\right)$ (see Eqs. () and ()), and define $\mathbf{n}\left(\mathbf{r}\right)$ as binarization noise with mean equal to zero. When the layer is wide, we expect the deterministic mean term $\mathrm{HT}\left(\mathbf{x}\right)$ to dominate, because the noise term $\mathbf{n}\left(\mathbf{r}\right)$ is a summation over many independent binarizations from all the neurons in the previous layer. Thus, we argue that the binarization noise $\mathbf{n}\left(\mathbf{x}\right)$ can be ignored when performing differentiation in the backward propagation stage. Therefore, we replace $\frac{\partial h_{b}\left(r\right)}{\partial r}$ (which cannot be computed) with $$\frac{\partial\mathrm{HT}\left(\mathbf{r}\right)}{\partial x}
     = \left\{ \begin{array}{ll}
            0 & \mbox{$r>1$},\\
            1 & \mbox{$r\in[-1,1]$},\\
            0 & \mbox{$r<-1$},\end{array} \right. \label{eq:BackProp}$$ which is exactly the straight-through estimator defined in Eq (). The use of this straight-through estimator is illustrated in Algorithm .

A similar binarization process was applied for weights in which we combine two ingredients:

-   Project each real-valued weight to \[-1,1\], i.e., clip the weights during training, as per Algorithm . The real-valued weights would otherwise grow very large without any impact on the binary weights.

-   When using a weight $w^r$, quantize it using $w^b = {\rm Sign}(w^r)$.

Projecting the weights to \[-1,1\] is consistent with the gradient cancelling when $|w^r|>1$, according to Eq. ( ).

## Shift-based Batch Normalization

Batch Normalization (BN) accelerates the training and reduces the overall impact of the weight scale . The normalization procedure may also help to regularize the model. However, at train-time, BN requires many multiplications (calculating the standard deviation and dividing by it, namely, dividing by the running variance, which is the weighted mean of the training set activation variance). Although the number of scaling calculations is the same as the number of neurons, in the case of ConvNets this number is quite large. For example, in the CIFAR-10 dataset (using our architecture), the first convolution layer, consisting of only $128\times3\times3$ filter masks, converts an image of size $3\times32\times32$ to size $128\times28\times28$, which is almost two orders of magnitude larger than the number of weights (87.1 to be exact). To achieve the results that BN would obtain, we use a shift-based batch normalization (SBN) technique, presented in Algorithm . SBN approximates BN almost without multiplications. Define $\mathrm{AP2}(z)$ as the approximate power-of-2 of z (i.e., the index of the most significant bit (MSB)), and $\ll\gg$ as both left and right binary shift. SBN replaces almost all multiplication with power-of-2-approximation and shift operations: $$x\times y\rightarrow x\ll\gg \mathrm{AP2}(y).$$

The only operation which is not a binary shift or an add is the inverse square root (see normalization operation Algorithm ). From the early work of we know that the inverse-square operation could be applied with approximately the same complexity as multiplication. There are also faster methods, which involve lookup table tricks that typically obtain lower accuracy (this may not be an issue, since our procedure already adds a lot of noise). However, the number of values on which we apply the inverse-square operation is rather small, since it is done after calculating the variance, i.e., after averaging (for a more precise calculation, see the BN analysis in . Furthermore, the size of the standard deviation vectors is relatively small. For example, these values make up only $0.3\%$ of the network size (i.e., the number of learnable parameters) in the Cifar-10 network we used in our experiments.

In the experiment we observed no loss in accuracy when using the shift-based BN algorithm instead of the vanilla BN algorithm.

## Shift Based AdaMax

The ADAM learning method also reduces the impact of the weight scale. Since ADAM requires many multiplications, we suggest using instead the shift-based AdaMax we outlined in Algorithm . In the experiment we conducted we observed no loss in accuracy when using the shift-based AdaMax algorithm instead of the vanilla ADAM algorithm.

## First Layer

In a BNN, only the binarized values of the weights and activations are used in all calculations. As the output of one layer is the input of the next, the inputs of all the layers are binary, with the exception of the first layer. However, we do not believe this to be a major issue. First, in computer vision, the input representation typically has far fewer channels (e.g, red, green and blue) than internal representations (e.g., 512). Consequently, the first layer of a ConvNet is often the smallest convolution layer, both in terms of parameters and computations . Second, it is relatively easy to handle continuous-valued inputs as fixed point numbers, with $m$ bits of precision. For example, in the common case of $8$-bit fixed point inputs: $$\label{eq:FirstLayer}
     s = x \cdot w^b, \qquad \qquad  \qquad\qquad     s = \sum_{n=1}^{8} {2^{n-1} (x^n \cdot w^b),}
    % s & = \sum_{i=1}^{1024}{x_i w^b_i} \\
    % s & = \sum_{i=1}^{1024}{(\sum_{n=1}^{8}{2^{n-1} x^n_i w^b_i})} \\
    % s & = \sum_{n=1}^{8} {(2^{n-1} \sum_{i=1}^{1024}{(x^n_i w^b_i)})} \\
    % s & = \sum_{n=1}^{8} {2^{n-1} (x^n \cdot w^b),}$$ where $x$ is a vector of 1024 8-bit inputs, $x_1^8$ is the most significant bit of the first input, $w^b$ is a vector of 1024 1-bit weights, and $s$ is the resulting weighted sum. This method is used in Algorithm .

# Qunatized Neural network - More than 1-bit

Observing Eq. (), we can see that using 2-bit activations simply doubles the number of times we need to run our XnorPopCount Kernel (i.e., directly proportional to the activation bitwidth). This idea was recently proposed by (DoReFa net) and (published on arXive shortly after our preliminary technical report was published there). However, in contrast to to , we did not find it useful to initialize the network with weights obtained by training the network with full precision weights. Moreover, the network did not quantize the weights of the first convolutional layer and the last fully-connected layer, whereas we binarized both. We followed the quantization schemes suggested by , namely, linear quantization: $$\mathbf{LinearQuant}(x, bitwidth)=\mathrm{Clip}\left(\mathrm{round}\left(\frac{x}{bitwidth}\right)\times bitwidth,minV,maxV\right)$$ and logarithmic quantization: $$\begin{aligned}
%&\tilde{x}=round\left(log_2(x)\right)\\
&\mathbf{LogQuant}(x, bitwidth)\left(\mathbf{x}\right) =\mathrm{Clip}\left(\mathrm{AP2}(x),minV,maxV\right),
\end{aligned}$$ where $minV$ and $maxV$ are the minimum and maximum scale range respectively. Where $\mathrm{AP2}(x)$ is the approximate-power-of-2 of $x$ as described in Section . In our experiments (detailed in Section ) we applied the above quantization schemes on the weights, activations and gradients and tested them on the more challenging ImageNet dataset.

# Benchmark Results

## Results on MNIST,SVHN, and CIFAR-10

```latex
\begin{table*}[ht]\protect\caption{Classification test error rates of DNNs trained on MNIST (fully connected architecture), CIFAR-10 and SVHN (convnet). No unsupervised pre-training or data augmentation was used.}
\centering{}%

\begin{tabular}{lcccr}
\hline
Data set  & MNIST & SVHN & CIFAR-10\tabularnewline
\hline
\hline
\multicolumn{5}{c}{Binarized activations+weights, during training and test}\tabularnewline
\hline
\hline
BNN (Torch7) & 1.40\% & 2.53\% & 10.15\% & \tabularnewline
BNN (Theano) & 0.96\% & 2.80\% & 11.40\% & \tabularnewline
% MC: I was under the impression that \citep{Baldassi2015} 's error rate was 1.25\%
% DS: Yes, but that was with RBM pre-training (non-binary!). I talked with Carlo, the author, and he said they got 1.35% without this.
Committee Machines' Array \cite{Baldassi2015} & 1.35\% & - & - \tabularnewline
\hline
\hline
\multicolumn{5}{c}{Binarized weights, during training and test}\tabularnewline
\hline
\hline
BinaryConnect \cite{Courbariaux-et-al-2015} & 1.29$\pm$ 0.08\% & 2.30\% & 9.90\% & \tabularnewline
\hline
\hline
\multicolumn{5}{c}{Binarized activations+weights, during test}\tabularnewline
\hline
\hline
EBP \cite{Cheng-et-al-2015} & 2.2$\pm$ 0.1\% & - & - & \tabularnewline
Bitwise DNNs \cite{Kim-et-al-2016} & 1.33\% & - & - & \tabularnewline
\hline
\hline
\multicolumn{5}{c}{Ternary weights, binary activations, during test}\tabularnewline
\hline
\hline
\cite{hwang-et-al-2014} & 1.45\% & - & - & \tabularnewline

\hline
\hline
\multicolumn{5}{c}{No binarization (standard results)}\tabularnewline
\hline
\hline
No reg & 1.3$\pm$ 0.2\% & 2.44\% & 10.94\% & \tabularnewline
Maxout Networks \cite{Goodfellow2013a} & 0.94\% & 2.47\% & 11.68\% & \tabularnewline
%Network in Network \citep{Lin} & - & 2.35\% & 10.41\% & \tabularnewline
Gated pooling \cite{lee-et-al-2015}& - & 1.69\% & 7.62\% & \tabularnewline
\hline
\end{tabular}\label{tab:benchmarks}

\end{table*}
```

We performed two sets of experiments, each based on a different framework, namely Torch7 and Theano. Other than the framework, the two sets of experiments are very similar:

-   In both sets of experiments, we obtain near state-of-the-art results with BNNs on MNIST, CIFAR-10 and the SVHN benchmark datasets.

-   In our Torch7 experiments, the activations are *stochastically* binarized at train-time, whereas in our Theano experiments they are *deterministically* binarized.

-   In our Torch7 experiments, we use the *shift-based BN and AdaMax* variants, which are detailed in Algorithms and , whereas in our Theano experiments, we use *vanilla BN and ADAM*.

Results are reported in Table . Implementation details are reported in Appendix A.

#### MNIST

MNIST is an image classification benchmark dataset . It consists of a training set of 60K and a test set of 10K 28 $\times$ 28 gray-scale images representing digits ranging from 0 to 9. The Multi-Layer-Perceptron (MLP) we train on MNIST consists of 3 hidden layers. In our Theano implementation we used hidden layers of size 4096 whereas in our Torch implementation we used much smaller size 2048. This difference explains the accuracy gap between the two implementations.

#### CIFAR-10

CIFAR-10 is an image classification benchmark dataset. It consists of a training set of size 50K and a test set of size 10K, where instances are 32 $\times$ 32 color images representing airplanes, automobiles, birds, cats, deer, dogs, frogs, horses, ships and trucks. Both implementations share the same structure as reported in Appendix A. Since the Torch implementation uses stochastic binarization, it achieved slightly better results.

#### SVHN

Street View House Numbers (SVHN) is also an image classification benchmark dataset. It consists of a training set of size 604K examples and a test set of size 26K, where instances are 32 $\times$ 32 color images representing digits ranging from 0 to 9. Here again we obtained a small improvement in the performance by using stochastic binarization scheeme.

## Results on ImageNet

To test the strength of our method, we applied it to the challenging ImageNet classification task, which is probably the most important classification benchmark dataset. It consists of a training set of size 1.2M samples and test set of size 50K. Each instance is labeled with one of 1000 categories including objects, animals, scenes, and even some abstract shapes. On ImageNet, it is customary to report two error rates: top-1 and top-5, where the top-$x$ error rate is the fraction of test images for which the correct label is not among the $x$ labels considered most probable by the model. Considerable research has been concerned with compressing ImageNet architectures while preserving high accuracy. Previous approaches include pruning near zero weights using matrix factorization techniques , quantizing the weights , using shared weights and applying Huffman codes among others.

To the best of our knowledge, before the first revision of this paper was published on arXive, no one had reported on successfully quantizing the network's activations. On the contrary, a recent work showed that accuracy significantly deteriorates when trying to quantize convolutional layers' weights below 4-bit (FC layers are more robust to quantization and can operate quite well with only 2 bits). In the present work we attempted to tackle the difficult task of binarizing both weights and activations. Employing the well-known AlexNet and GoogleNet architectures, we applied our techniques and achieved $41.8\%$ top-1 and $67.1\%$ top-5 accuracy using AlexNet and $47.1\%$ top-1 and $69.1\%$ top-5 accuracy using GoogleNet. While these performance results leave room for improvement (relative to full precision nets), they are by far better than all previous attempts to compress ImageNet architectures using less than 4-bit precision for the weights. Moreover, this advantage is achieved while also binarizing neuron activations.

## Relaxing "hard tanh"$$ boundaries

We discovered that after training the network it is useful to widen the "hard tanh"$$ boundaries and retrain the network. As explained in Section , the straight-through estimator (which can be written as "hard tanh") cancels gradients coming from neurons with absolute values higher than 1. Hence, towards the last training iterations most of the gradient values are zero and the weight values cease to update. By relaxing the "hard tanh"$$ boundaries we allow more gradients to flow in the back-propagation phase and improve top-1 accuracies by $1.5\%$ on AlexNet topology using vanilla BNN.

## 2-bit activations

While training BNNs on the ImageNet dataset we noticed that we could not force the training set error rate to converge to zero. In fact the training error rate stayed fairly close to the validation error rate. This observation led us to investigate a more relaxed activation quantization (more than 1-bit). As can be seen in Table , the results are quite impressive and illustrate an approximate $5.6\%$ drop in performance (top-1 accuracy) relative to floating point representation, using only 1-bit weights and 2-bit activation. Following , we also tried quantizing the gradients and discovered that only logarithmic quantization works. With 6-bit gradients we achieved $46.8\%$ degradation. Those results are presently state-of-the-art, surpassing those obtained by the DoReFa net . As opposed to DoReFa, we utilized a deterministic quantization process rather than a stochastic one. Moreover, it is important to note that while quantizing the gradients, DoReFa assigns for each instance in a mini-batch its own scaling factor, which increases the number of MAC operations.

While AlexNet can be compressed rather easily, compressing GoogleNet is much harder due to its small number of parameters. When using vanilla BNNs, we observed a large degradation in the top-1 results. However, by using QNNs with 4-bit weights and activation, we were able to achieve $66.5\%$ top-1 accuracy (only a $5.5\%$ drop in performance compared to the 32-bit floating point architecture), which is the current state-of-the-art-compression result over GoogleNet. Moreover, by using QNNs with 6-bit weights, activations and gradients we achieved $66.4\%$ top-1 accuracy. Full implementation details of our experiments are reported in Appendix .

```latex
\begin{table*}[ht]\protect\caption{Classification test error rates of the AlexNet model trained on the ImageNet 1000 classification task. No unsupervised pre-training or data augmentation was used.}
\centering{}%
\scalebox{1}{
\begin{tabular}{lcccr}
\hline
Model & Top-1 & Top-5 &&\tabularnewline
\hline
\hline
\multicolumn{5}{c}{Binarized activations+weights, during training and test}\tabularnewline
\hline
\hline
BNN  & 41.8\% & 67.1\% & \tabularnewline
Xnor-Nets\tablefootnote{\label{first_last} First and last layers were not binarized (i.e., using 32-bit precision weights and activation)} ~\citep{rastegari2016xnor} & 44.2\% & 69.2\%   \tabularnewline
\hline
\hline
\multicolumn{5}{c}{Binary weights and Quantize activations during training and test}\tabularnewline
QNN 2-bit activation  & 51.03\% & 73.67\% &  \tabularnewline
DoReFaNet 2-bit activation\footref{first_last} \citep{zhou2016dorefa} & 50.7\% & 72.57\% & \tabularnewline
\hline
\hline
\multicolumn{5}{c}{Quantize weights, during test}\tabularnewline
\hline
\hline
Deep Compression 4/2-bit (conv/FC layer) \citep{han2015deep} & 55.34\% & 77.67\% &  \tabularnewline
\citep{gysel2016hardware} - 2-bit  & 0.01\% & -\% &  \tabularnewline
\hline
\hline
\multicolumn{5}{c}{No Quantization (standard results)}\tabularnewline
\hline
\hline
AlexNet - our implementation & 56.6\% & 80.2\% &  \tabularnewline
\hline
\end{tabular}\label{tab:AlexNet}
}
\end{table*}
```
```latex
\begin{table*}[h]\protect\caption{Classification test error rates of the GoogleNet model trained on the ImageNet 1000 classification task. No unsupervised pre-training or data augmentation was used.}
\centering{}%
\scalebox{1}{
\begin{tabular}{lcccr}
\hline
Model  & Top-1 & Top-5 &&\tabularnewline
\hline
\hline
\multicolumn{5}{c}{Binarized activations+weights, during training and test}\tabularnewline
\hline
\hline
BNN  & 47.1\% & 69.1\% & \tabularnewline
\hline
\multicolumn{5}{c}{Quantize weights and activations during training and test}\tabularnewline
\hline
\hline
QNN 4-bit & 66.5\% & 83.4\% & \tabularnewline
\hline
\hline
\multicolumn{5}{c}{Quantize activation,weights and gradients during training and test}\tabularnewline
\hline
\hline
QNN 6-bit & 66.4\% & 83.1\% &
\tabularnewline
\hline
\hline
\multicolumn{5}{c}{No Quantization (standard results)}\tabularnewline
\hline
\hline
GoogleNet - our implementation & 71.6\% & 91.2\% &  \tabularnewline
\hline
\end{tabular}\label{tab:GoogleNet}
}
\end{table*}
```

## Language Models

Recurrent neural networks (RNNs) are very demanding in memory and computational power in comparison to feed forward networks. There are a large variety of recurrent models with the Long Short Term Memory networks (LSTMs) introduced by are being the most popular model. LSTMs are a special kind of RNN, capable of learning long-term dependencies using unique gating mechanisms. Recently, tried to quantize the RNNs weight matrices using similar techniques as described in Section . They observed that the weight binarization methods do not work with RNNs. However, by using 2-bits (i.e., ${-1,0,1}$), they have been able to achieve similar and even higher accuracy on several datasets. Here we report on the first attempt to quantize both weights and activations by trying to evaluate the accuracy of quantized recurrent models trained on the Penn Treebank dataset. The Penn Treebank Corpus contains 10K unique words. We followed the same setting as in which resulted in 18.55K words for training set, 14.5K and 16K words in the validation and test sets respectively. We experimented with both vanilla RNNs and LSTMs. For our vanilla RNN model we used one hidden layers of size 2048 and ReLU as the activation function. For our LSTM model we use 1 hidden layer of size 300. Our RNN implementation was constructed to predict the next character hence performance was measured using the bits-per-character (BPC) metric. In the LSTM model we tried to predict the next word so performance was measured using the perplexity per word (PPW) metric. Similar to , our preliminary results indicate that binarization of weight matrices lead to large accuracy degradation. However, as can be seen in Table , with 4-bits activations and weights we can achieve similar accuracies as their 32-bit floating point counterparts.

```latex
\begin{table*}[h]\protect\caption{Language Models results on Penn Treebank dataset.}{Language Models results on Penn Treebank dataset. FP stands for 32-bit floating point}
\centering{}%
\scalebox{1}{
\begin{tabular}{lccccccr}
\hline
Model  & Layers & Hidden Units & bits(weights) & bits(activation) & Accuracy &&\tabularnewline
\hline
\hline
RNN    & 1 & 2048 & 3  & 3  &1.81 BPC &  \tabularnewline
RNN    & 1 & 2048 & 2  & 4  &1.67  BPC &  \tabularnewline
RNN    & 1 & 2048 & 3  & 4  &1.11 BPC &  \tabularnewline
RNN    & 1 & 2048 & 3  & 4  &1.05 BPC & \tabularnewline
RNN    & 1 & 2048 & FP & FP &1.05 BPC & \tabularnewline
\hline
\hline
LSTM    & 1 & 300 & 2  & 3  & 220 PPW &  \tabularnewline
LSTM    & 1 & 300 & 3  & 4  & 110 PPW&  \tabularnewline
LSTM    & 1 & 300 & 4  & 4  & 100 PPW&  \tabularnewline
LSTM    & 1 & 900 & 4  & 4  & 97 PPW&  \tabularnewline
LSTM    & 1 & 300 & FP & FP & 97 PPW&  \tabularnewline
\end{tabular}\label{tab:LangModels}
}
\end{table*}
```

# High Power Efficiency during the Forward Pass

| Operation             | MUL   |  ADD   |
|:----------------------|:------|:------:|
| 8-bit Integer         | 0.2pJ | 0.03pJ |
| 32-bit Integer        | 3.1pJ | 0.1pJ  |
| 16-bit Floating Point | 1.1pJ | 0.4pJ  |
| 32-bit Floating Point | 3.7pJ | 0.9pJ  |

Energy consumption of memory accesses; see

| Memory size | 64-bit Cache |     |
|:------------|:-------------|:---:|
| 8K          | 10pJ         |     |
| 32K         | 20pJ         |     |
| 1M          | 100pJ        |     |
| DRAM        | 1.3-2.6nJ    |     |

Energy consumption of memory accesses; see

Computer hardware, be it general-purpose or specialized, is composed of memories, arithmetic operators and control logic. During the forward pass (both at run-time and train-time), BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which might lead to vastly improved power-efficiency. Moreover, a binarized CNN can lead to binary convolution kernel repetitions, and we argue that dedicated hardware could reduce the time complexity by $60\%$ .

#### Memory Size and Accesses

Improving computing performance has always been and remains a challenge. Over the last decade, power has been the main constraint on performance . This is why considerable research efforts have been devoted to reducing the energy consumption of neural networks. provides rough numbers for the energy consumed by the computation (the given numbers are for 45nm technology), as summarized in Tables and . Importantly, we can see that memory accesses typically consume more energy than arithmetic operations, and *memory access cost increases with memory size*. In comparison with 32-bit DNNs, BNNs require 32 times smaller memory size *and* 32 times fewer memory accesses. This is expected to reduce energy consumption drastically (i.e., by a factor larger than 32).

#### XNOR-Count

Applying a DNN mainly involves convolutions and matrix multiplications. The key arithmetic operation of deep learning is thus the multiply-accumulate operation. Artificial neurons are basically multiply-accumulators computing weighted sums of their inputs. In BNNs, both the activations and the weights are constrained to either $-1$ or $+1$. As a result, most of the 32-bit floating point multiply-accumulations are replaced by 1-bit XNOR-count operations. This could have a big impact on dedicated deep learning hardware. For instance, a 32-bit floating point multiplier costs about 200 Xilinx FPGA slices , whereas a 1-bit XNOR gate only costs a single slice.

When using a ConvNet architecture with binary weights, the number of unique filters is bounded by the filter size. For example, in our implementation we use filters of size $3\times3$, so the maximum number of unique 2D filters is $2^{9}=512$. However, this should not prevent expanding the number of feature maps beyond this number, since the actual filter is a 3D matrix. Assuming we have $M_{\ell}$ filters in the $\ell$ convolutional layer, we have to store a 4D weight matrix of size $M_{\ell}\times M_{\ell-1}\times k\times k$. Consequently, the number of unique filters is $2^{k^{2}M_{\ell-1}}$. When necessary, we apply each filter on the map and perform the required multiply-accumulate (MAC) operations (in our case, using XNOR and popcount operations). Since we now have binary filters, many 2D filters of size $k\times k$ repeat themselves. By using dedicated hardware/software, we can apply only the unique 2D filters on each feature map and sum the results to receive each 3D filter's convolutional result. Note that an inverse filter (i.e., \[-1,1,-1\] is the inverse of \[1,-1,1\]) can also be treated as a repetition; it is merely a multiplication of the original filter by -1. For example, in our ConvNet architecture trained on the CIFAR-10 benchmark, there are only 42% unique filters per layer on average. Hence we can reduce the number of the XNOR-popcount operations by 3.

QNNs complexity scale up linearly with the number of bits per weight/activation, since it requires the application of the XNOR kernel several times (see Section ). As of now, QNNs still supply the best compression to accuracy ratio. Moreover, quantizing the gradients allows us to use the XNOR kernel for the backward pass, leading to fully fixed point layers with low bitwidth. By accelerating the training phase, QNNs can play an important role in future power demanding tasks.

# Seven Times Faster on GPU at Run-Time

It is possible to speed up GPU implementations of QNNs, by using a method sometimes called SIMD (single instruction, multiple data) within a register (SWAR). The basic idea of SWAR is to *concatenate* groups of 32 binary variables into 32-bit registers, and thus obtain a 32-times speed-up on bitwise operations (e.g., XNOR). Using SWAR, it is possible to evaluate 32 connections with only 3 instructions: $$\begin{aligned}
    \label{eq:popc}
    a_1 += {\rm popcount(xnor}(a_0^{32b},w^{32b}_1)),
\end{aligned}$$ where $a_1$ is the resulting weighted sum, and $a_0^{32b}$ and $w^{32b}_1$ are the concatenated inputs and weights. Those 3 instructions (accumulation, popcount, xnor) take $1+4+1=6$ clock cycles on recent Nvidia GPUs (and if they were to become a fused instruction, it would only take a single clock cycle). Consequently, we obtain a theoretical Nvidia GPU speed-up of factor of $32/6 \approx 5.3$. In practice, this speed-up is quite easy to obtain as the memory bandwidth to computation ratio is also increased 6 times.

In order to validate those theoretical results, we programmed two GPU kernels:

-   An unoptimized matrix multiplication kernel that serves as our baseline.

-   The XNOR kernel, which is nearly identical to the baseline, except that it uses the SWAR method, as in Equation ().

The two GPU kernels return identical outputs when their inputs are constrained to $-1$ or $+1$ (but not otherwise). The XNOR kernel is about *23 times faster than the baseline kernel* and *3.4 times faster than cuBLAS*, as shown in Figure . Last but not least, the MLP from Section runs 7 times faster with the XNOR kernel than with the baseline kernel, without suffering any loss in classification accuracy (see Figure ). As MNIST's images are not binary, the first layer's computations are always performed by the baseline kernel. The last three columns show that the MLP accuracy does not depend on which kernel is used.

# Discussion and Related Work

Until recently, the use of extremely low-precision networks (binary in the extreme case) was believed to substantially degrade the network performance . and proved the contrary by showing that good performance could be achieved even if all neurons and weights are binarized to $\pm 1$ . This was done using Expectation BackPropagation (EBP), a variational Bayesian approach, which infers networks with binary weights and neurons by updating the posterior distributions over the weights. These distributions are updated by differentiating their parameters (e.g., mean values) via the back propagation (BP) algorithm. implemented a fully binary network at run time using a very similar approach to EBP, showing significant improvement in energy efficiency. The drawback of EBP is that the binarized parameters are only used during inference.

The probabilistic idea behind EBP was extended in the BinaryConnect algorithm of . In BinaryConnect, the real-valued version of the weights is saved and used as a key reference for the binarization process. The binarization noise is independent between different weights, either by construction (by using stochastic quantization) or by assumption (a common simplification; see ). The noise would have little effect on the next neuron's input because the input is a summation over many weighted neurons. Thus, the real-valued version could be updated using the back propagated error by simply ignoring the binarization noise in the update. With this method, were the first to binarize weights in CNNs and achieved near state-of-the-art performance on several datasets. They also argued that noisy weights provide a form of regularization, which could help to improve generalization, as previously shown by . This method binarized weights while still maintaining full precision neurons.

carried over the work of to the back-propagation process by quantizing the representations at each layer of the network, to convert some of the remaining multiplications into binary shifts by restricting the neurons' values to be power-of-two integers. 's work and ours seem to share similar characteristics .However, their approach continues to use full precision weights during the test phase. Moreover, quantize the neurons only during the back propagation process, and not during forward propagation.

Other research showed that full binary training and testing is possible in an array of committee machines with randomized input, where only one weight layer is being adjusted. aimed to compress a fully trained high precision network by using quantization or matrix factorization methods. These methods required training the network with full precision weights and neurons, thus requiring numerous MAC operations (which the proposed QNN algorithm avoids). focused on a fixed-point neural network design and achieved performance almost identical to that of the floating-point architecture. *retrained* neural networks with binary weights and activations.

As far as we know, before the first revision of this paper was published on arXive, no work succeeded in binarizing weights *and* neurons, at the inference phase *and* the entire training phase of a deep network. This was achieved in the present work. We relied on the idea that binarization can be done stochastically, or be approximated as random noise. This was previously done for the weights by , but our BNNs extend this to the activations. Note that the binary activations are especially important for ConvNets, where there are typically many more neurons than free weights. This allows highly efficient operation of the binarized DNN at run time, and at the forward-propagation phase during training. Moreover, our training method has almost no multiplications, and therefore might be implemented efficiently in dedicated hardware. However, we have to save the value of the full precision weights. This is a remaining computational bottleneck during training, since it is an energy-consuming operation.

Shortly after the first version of this paper was posted on arXiv, several papers tried to improve and extend it. made a small modification to our algorithm (namely multiplying the binary weights and input by their $L_1$ norm) and published promising results on the ImageNet dataset. Note that their method, named Xnor-Net, requires additional multiplication by a different scaling factor for each patch in each sample Section 3.2 Eq. 10 and figure 2). This in itself, requires many multiplications and prevents efficient implementation of XnorNet on known hardware designs. Moreover, didn't quantize first and last layers, therefore XNOR-Net are only partially binarized NNs. suggested a more relaxed quantization (more than 1-bit) for both the weights and activation. Their idea was to quantize both and use shift operations as in our Eq. (). They proposed to quantize the parameters in their non-uniform, base-2 logarithmic representation. This idea was inspired by the fact that the weights and activations in a trained network naturally have non-uniform distributions. They moreover showed that they can quantize the gradients as well to 6-bit without significant losses in performance (on the Cifar-10 dataset). applied similar ideas to the ImageNet dataset and showed that by using 1-bit weights, 2-bit activations and 6-bit gradients they can achieve $46.1\%$ top-1 accuracies using the AlexNet architecture. They named this method DoReFa net. Here we outperform DoReFa net and achieve $46.8\%$ using a 1-2-6 bit quantization scheme (weight-activation-gradients) and $51\%$ using a 1-2-32 quantization scheme. These results confirm that we can achieve comparable results even on a large dataset by applying the Xnor kernel several times. showed that DNN can be robust to more than just weight binarization. They applied several different distortions to the weights, including additive and multiplicative noise, and a class of non-linear projections.This was shown to improve robustness to other distortions and even boost results. tried to apply our binarization scheme to recurrent neural network for language modeling and achieved comparable results as well. even created a hardware implementation to speed up BNNs.

# Conclusion

We have introduced BNNs, which binarize deep neural networks and can lead to dramatic improvements in both power consumption and computation speed. During the forward pass (both at run-time and train-time), BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations. Our estimates indicate that power efficiency can be improved by more than one order of magnitude (see Section ). In terms of speed, we programmed a binary matrix multiplication GPU kernel that enabled running MLP over the MNIST datset 7 times faster (than with an unoptimized GPU kernel) without any loss of accuracy (see Section ).

We have shown that BNNs can handle MNIST, CIFAR-10 and SVHN while achieving nearly state-of-the-art accuracy. While our results for the challenging ImageNet are not on par with the best results achievable with full precision networks, they significantly improve all previous attempts to compress ImageNet-capable architectures. Moreover, by quantizing the weights and activations to more than 1-bit (i.e., QNNs), we have been able to achieve comparable results to the 32-bit floating point architectures (see Section  and supplementary material - Appendix B). A major open research avenue would be to further improve our results on ImageNet. Substantial progress in this direction might go a long way towards facilitating DNN usability in low power instruments such as mobile phones.

# Implementation Details

In this section we give full implementation details over our MNIST,SVHN, CIFAR-10 and ImageNet datasets.

## MLP on MNIST (Theano)

MNIST is an image classification benchmark dataset . It consists of a training set of 60K and a test set of 10K 28 $\times$ 28 gray-scale images representing digits ranging from 0 to 9. In order for this benchmark to remain a challenge, we did not use any convolution, data-augmentation, preprocessing or unsupervised learning. The Multi-Layer-Perceptron (MLP) we train on MNIST consists of 3 hidden layers of 4096 binary units and a L2-SVM output layer; L2-SVM has been shown to perform better than Softmax on several classification benchmarks . We regularize the model with Dropout . The square hinge loss is minimized with the ADAM adaptive learning rate method . We use an exponentially decaying global learning rate, as per Algorithm 1, and also scale the learning rates of the weights with their initialization coefficients from , as suggested by . We use Batch Normalization with a minibatch of size 100 to speed up the training. As is typical, we use the last 10K samples of the training set as a validation set for early stopping and model selection. We report the test error rate associated with the best validation error rate after 1000 epochs (we do not retrain on the validation set).

## MLP on MNIST (Torch7)

We use a similar architecture as in our Theano experiments, without dropout, and with 2048 binary units per layer instead of 4096. Additionally, we use the shift base AdaMax and BN (with a minibatch of size 100) instead of the vanilla implementations, to reduce the number of multiplications. Likewise, we decay the learning rate by using a 1-bit right shift every 10 epochs.

## ConvNet on CIFAR-10 (Theano)

CIFAR-10 is an image classification benchmark dataset. It consists of a training set of size 50K and a test set of size 10K, where instances are 32 $\times$ 32 color images representing airplanes, automobiles, birds, cats, deer, dogs, frogs, horses, ships and trucks. We do not use data-augmentation (which can really be a game changer for this dataset; see ). The architecture of our ConvNet is identical to that used by except for the binarization of the activations. The architecture is itself mainly inspired by VGG . The square hinge loss is minimized with ADAM. We use an exponentially decaying learning rate, as we did for MNIST. We scale the learning rates of the weights with their initialization coefficients from . We use Batch Normalization with a minibatch of size 50 to speed up the training. We use the last 5000 samples of the training set as a validation set. We report the test error rate associated with the best validation error rate after 500 training epochs (we do not retrain on the validation set).

```latex
\scalebox{.75}{
\begin{tabular}{@{}l@{}}
\toprule
CIFAR-10 ConvNet architecture                  \\ \midrule
Input: $32\times32$ - RGB image                         \\
$3\times3$ - 128 convolution layer                      \\
BatchNorm and Binarization layers              \\
$3\times3$ - 128 convolution and $2\times2$ max-pooling layers \\
BatchNorm and Binarization layers              \\ \midrule
$3\times3$ - 256 convolution layer                      \\
BatchNorm and Binarization layers              \\
$3\times3$ - 256 convolution and $2\times2$ max-pooling layers \\
BatchNorm and Binarization layers              \\ \midrule
$3\times3$ - 512 convolution layer                      \\
BatchNorm and Binarization layers              \\
$3\times3$ - 512 convolution and $2\times2$ max-pooling layers \\
BatchNorm and Binarization layers              \\ \midrule
1024 fully connected layer                     \\
BatchNorm and Binarization layers              \\
1024 fully connected layer                     \\
BatchNorm and Binarization layers              \\ \midrule
10 fully connected layer                       \\
BatchNorm layer (no binarization)              \\
Cost: Mean square hinge loss                   \\ \bottomrule
\end{tabular}
}
```

## ConvNet on CIFAR-10 (Torch7)

We use the same architecture as in our Theano experiments. We apply shift-based AdaMax and BN (with a minibatch of size 200) instead of the vanilla implementations to reduce the number of multiplications. Likewise, we decay the learning rate by using a 1-bit right shift every 50 epochs.

## ConvNet on SVHN

SVHN is also an image classification benchmark dataset. It consists of a training set of size 604K examples and a test set of size 26K, where instances are 32 $\times$ 32 color images representing digits ranging from 0 to 9. In both sets of experiments, we follow the same procedure used for the CIFAR-10 experiments, with a few notable exceptions: we use half the number of units in the convolution layers, and we train for 200 epochs instead of 500 (because SVHN is a much larger dataset than CIFAR-10).

## ConvNet on ImageNet

ImageNet classification task consists of a training set of size 1.2M samples and test set of size 50K. Each instance is labeled with one of 1000 categories including objects, animals, scenes, and even some abstract shapes.

#### AlexNet:

Our AlexNet implementation consists of 5 convolution layers followed by 3 fully connected layers (see Section ). Additionally, we use Adam as our optimization method and batch-normalization layers (with a minibatch of size 512). Likewise, we decay the learning rate by 0.1 every 20 epochs.

#### GoogleNet:

Our GoogleNet implementation consist of 2 convolution layers followed by 10 inception layers, spatial-average-pooling and a fully connected classifier. We also used the 2 auxilary classifiers. Additionally, we use Adam as our optimization method and batch-normalization layers (with a minibatch of size 64). Likewise, we decay the learning rate by 0.1 every 10 epochs.

```latex
\scalebox{.75}{
\begin{tabular}{@{}l@{}}
\toprule
AlexNet ConvNet architecture                  \\ \midrule
Input: $32\times32$ - RGB image                         \\
$11\times11$ - 64 convolution layer and $3\times3$ max-pooling layers   \\
BatchNorm and Binarization layers              \\
$5\times5$ - 192 convolution layer   and $3\times3$ max-pooling layers  \\
BatchNorm and Binarization layers              \\
$3\times3$ - 384 convolution layer                      \\
BatchNorm and Binarization layers              \\
$3\times3$ - 256 convolution layer  \\
BatchNorm and Binarization layers              \\
$3\times3$ - 256 convolution layer                      \\
BatchNorm and Binarization layers              \\
4096 fully connected layer                     \\
BatchNorm and Binarization layers              \\
4096 fully connected layer                     \\
BatchNorm and Binarization layers              \\ \midrule
1000 fully connected layer                       \\
BatchNorm layer (no binarization) \\
SoftMax layer (no binarization) \\
Cost: Negative log likelihood                   \\ \bottomrule
\end{tabular}
}
```

[^1]: <https://github.com/MatthieuCourbariaux/BinaryNet>

[^2]: <https://github.com/itayhubara/BinaryNet>


author: David Schneider; Kathleen McCoy
Departments of Linguistics and CIS
University of Delaware
Newark, DE 19716
{dschneid, email@example.com
bibliography: /usa/mccoy/papers/treffile.bib; /usa/suri/Bibliography/big.bib
title: Recognizing Syntactic Errors in the
  Writing of Second Language Learners[^1]

# Overview

This paper reports on the error-recognition component of the ICICLE (Interactive Computer Identification and Correction of Language Errors) system. The system is designed to be a tutorial system for helping second-language (L2) learners of English. In this instantiation of the system, we are focusing on the particular problems of American Sign Language (ASL) native signers. The system recognizes errors by using mal-rules (also called 'error-production rules') , which extend the language accepted by the grammar to include sentences containing the specified errors. The mal-rules themselves are derived from an error taxonomy which was the result of an analysis of writing samples. This paper focuses primarily on the unique challenges posed by developing a grammar that allows the parser to efficiently parse and recognize errors in sentences even when multiple errors occur. Additionally, it is important to note that the users will not be at a uniform stage of acquisition $-$ the system must be capable of processing the input of users with varying levels of English competence. We briefly describe how acquisition is modeled and how this model can help with some of the problems faced by a system designed to recognize errors.

We will begin with an overview of the entire ICICLE system. To motivate some of the difficulties encountered by our mal-rule-based error recognition system, we will briefly describe some of the errors common to the population under study. A major problem that must be faced is parsing efficiency caused by multiple parses. This is a particularly difficult problem when expected errors include omission errors, and thus this class of errors will be discussed in some detail. Another important problem involves the addition/subtraction of various syntactic features in the grammar and lexicon during acquisition. We describe how our system models this without the use of multiple lexicons. We follow this by a description of the current implementation and grammar coverage of the system. Finally, we will present an evaluation of the system for number/agreement errors in the target group of language learners.

# System Overview

The ICICLE system is meant to help second-language learners by identifying errors and engaging the learners in a tutorial dialogue. It takes as input a text written by the student. This is given to the error identification component, which is responsible for flagging the errors. The identification is done by parsing the input one sentence at a time using a bottom-up chart parser which is a successor to . The grammar formalism used by the parser consists of context-free rules augmented with features. The grammar itself is a grammar of English which has been augmented with a set of mal-rules which capture errors common to this user population. We will briefly discuss some classes of errors that were uncovered in our writing sample analysis which was used to identify errors expected in this population. This discussion will motivate some of the mal-rules which were written to capture some classes of errors, and the difficulties encountered in implementing these mal-rules. The mal-rules are specially tagged with information helpful in the correction phase of the system.

The error identification component relies on information in the user model $-$ the most interesting aspect of which is a model of the acquisition of a second language. This model (instantiated with information from the ASL/English language model) is used to highlight those grammar rules which the student has most likely already acquired or is currently in the process of acquiring. These rules will be the ones the parser attempts to use when parsing the user's input. Thus we take an interlanguage view of the acquisition process , , and attempt to model how the student's grammar is likely to change over time. The essence of the acquisition model is that there are discrete stages that all learners of a particular language will go through , , , . Each of these stages is characterized in our model by sets of language features (and therefore constructions) that the learner is in the process of acquiring. It is anticipated that most of the errors that learners make will be within the constructions (where "construction" is construed broadly) that they are in the process of acquiring and that they will favor sentences involving those constructions in a "hypothesize and test" style of learning, as predicted by interlanguage theory. Thus, the parser favors grammar rules involving constructions currently being acquired (and, to a lesser extent, constructions already acquired).

The correction phase of the system is a focus of current research. A description of the strategies for this phase can be found in and .

# Expected Errors

In order to identify the errors we expect the population to make, we collected writing samples from a number of different schools and organizations for the deaf. To help identify any instances of language transfer between ASL and written English, we concentrated on eliciting samples from deaf people who are native ASL signers. It is important to note that ASL is not simply a translation of standard English into manual gestures, but rather is a complete language with its own syntax, which is significantly different from English. Some of our previous work explored how language transfer might influence written English and suggested that negative language transfer might occur when the realization of specific language features differed between the first language and written English. For instance, one feature is the realization of the copula "be". In ASL the copula "be" is often not lexicalized. Thus, negative language transfer might predict omission errors resulting from not lexicalizing the copula "be" in the written English of ASL signers. While we concentrate here on errors from the ASL population, the errors identified are likely to be found in learners coming from first languages other than ASL as well. This would be the case if the first language has features in common with ASL. For instance the missing copula "be" is also a common error in the writing of native Chinese speakers since Chinese and ASL share the feature that the copula "be" is often not lexicalized. Thus, the examples seen here will generalize to other languages.

In the following we describe some classes of errors which we uncovered (and attempt to "explain" why an ASL native might come to make these errors).

## Constituent Omissions

Learners of English as a second language (ESL) omit constituents for a variety of reasons. One error that is common for many ASL learners is the dropping of determiners. Perhaps because ASL does not have a determiner system similar to that of English, it is not unusual for a determiner to be omitted as in:

** 1**.  * I am `_` transfer student from .... *

These errors can be flagged reasonably well when they are syntactic (and not pragmatic) in nature and do not pose much additional burden on the parser/grammar.

However, missing main verbs (most commonly missing copulas) are also common in our writing samples:

** 2**.  * Once the situation changes they `_` different people. *

One explanation for this (as well as other missing elements such as missing prepositions) is that copulas are not overtly lexicalized in ASL because the copula (preposition) is gotten across in different ways in ASL. Because the copula (preposition) is realized in a radically different fashion in ASL, there can be no positive language transfer for these constructions.

In addition to omitting verbs, some NPs may also be omitted. It has been argued (see, for example ) that ASL allows topic NP deletion which means that topic noun phrases that are prominent in the discourse context may be left out of a sentence. Carrying this strategy over to English might explain why some NPs are omitted from sentences such as:

** 3**.  * While living at college I spend lot of money because `_` go out to eat almost everyday. *

Mal-rules written to handle these errors must capture missing verbs, NPs, and prepositions. The grammar is further complicated because ASL natives also have many errors in relative clause formation including missing relative pronouns. The possibility of all of these omissions causes the parser to explore a great number of parses (many of which will complete successfully).

## Handling Omissions

As we just saw, omissions are frequent in the writing of ASL natives and they are difficult to detect using the mal-rule formalism. To clearly see the problem, consider the following two sentences, which would not be unusual in the writing of an ASL native.

** 4**.  * The boy happy. *

** 5**.  * Is happy. *

As the reader can see, in the main verb "be" is omitted, while the subject is missing in .

To handle these types of sentences, we included in our grammar mal-rules like the following:

** 6**.  * `VP(error +) `$\rightarrow$` AdjP` *

** 7**.  * `S(error +) `$\rightarrow$` VP` *

A significant problem that arises from these rules is that a simple adjective is parsed as an S even if it is in a normal, grammatical sentence. This behavior leads to many extra parses, since the S will be able to participate in lots of other parses. The problem becomes much more serious when the other possible omissions are added into the grammar. However, closer examination of our writing samples indicates that, except for determiners, our users generally leave out at most one word (constituent) per sentence. Thus it is unlikely that "happy" will ever be an entire sentence. We would like this fact to be reflected in the analyses explored by the parser. However, a traditional bottom-up context-free parser has no way to deal with this case, as there is no way to block rules from firing as long as the features are capable of unification.

One possibility would be to allow the `(error +)` feature to percolate up through the parse. Any rule which introduces the ` (error +)` feature could then be prevented from having any children specified with `(error +)`. However, this solution would be far too restrictive, as it would restrict the number of errors in a sentence to one, and many of the sentences in our ASL corpus involve multiple errors.

Recall, however, that in our analysis we found that (except for determiners) our writing samples did not contain multiple omission errors in a sentence. Thus another possibility might be to percolate an error feature associated with omissions only$-$perhaps called ` (missing +)`.

Upon closer inspection, this solution also has difficulties. The first difficulty has to do with implementing the feature percolation. For instance, for a VP to be specified as `(missing +)` whenever any of its sub-constituents has that feature, one would need to have separate rules raising the feature up from each of the sub-constituents, as in the following:

** 8**.  * `VP(missing ?a) `$\rightarrow$` V NP NP(missing ?a)` *

** 9**.  * `VP(missing ?a) `$\rightarrow$` V NP(missing ?a) NP` *

** 10**.  * `VP(missing ?a) `$\rightarrow$` V(missing ?a) NP NP` *

This would cause an unwarranted increase in the size of the grammar, and would also cause an immense increase in the number of parses, since three VPs would be added to the chart, one for each of the rules.

At first glance it appears that this problem can be overcome with the use of "foot features," which are included in the parser we are using. A foot feature moves features from any child to the parent. For example, for a foot feature F, if one child has a specification for F, it will be passed on to the parent. If more than one child is specified for F, then the values of F must unify, and the unified value will be passed up the parent. While the use of foot features appears to make the feature percolation easier, it will not allow the feature to be used as desired. In particular, we need to have the feature percolated *only* when it has a positive value and only when that value is associated with exactly one constituent on the right-hand side of a rule. The foot feature as defined by the parser would allow the percolation of the feature even if it were specified in more than one constituent.

A further complication with using this type of feature propagation arises because there are some situations where multiple omission errors do occur, especially when determiners are omitted.[^2] Consider the following example taken from our corpus where both the main verb "be" and a determiner "the" are omitted.

** 11**.  * Student always bothering me while I am at dorm.
(Corrected) Students are always bothering me while I am at the dorm. *

Our solution to the problem involves using procedural attachment. The parser we are using builds constituents and stores them in a chart. Before storing them in the chart, the parser can run arbitrary procedures on new constituents. These procedures, specified in the grammar, will be run on all constituents that meet a certain pattern specified by the grammar writer.

Our procedure amounts to specifying an alternative method for propagating the `(missing +)` feature, which will still be a foot feature. It will be run on any constituent that specifies `(missing +)`. The procedure can either delete a constituent that has more than one child with `(missing +)`, or it can alter the `(missing +)` feature on the constituent in the face of determiner omissions (as discussed in footnote ). By using a special procedure to implement the feature percolation, we will be able to be more flexible in where we allow the "missing" feature to percolate.

## Syntactic Feature Addition

For this system to properly model language acquisition, it must also model the addition (and possible subtraction) of syntactic features in the lexicon and grammar of the learner. For instance, ASL natives have a great deal of difficulty with many of the agreement features in English. As a concrete example, this population frequently has trouble with the difference between "other" and "another". They frequently use "other" in a singular NP, where "another" would normally be called for. We hypothesize that this is partly a result of their not understanding that there is agreement between NPs and their specifiers (determiners, quantifiers, etc.). Even if this is recognized, the learners may not have the lexical representations necessary to support the agreement for these two words.[^3] Thus, the most accurate model of the language of these early learners involves a lexicon with impoverished entries $-$ i.e. no person or number features for determiners and quantifiers. Such an impoverished lexicon would mean that the entries for the two words might be identical, which appears to be the case for these learners.

There are at least two reasons for not using this sort of impoverished lexicon. Firstly, it would require having multiple lexicons (some impoverished, others not), with the system needing to determine which to use for a given user. Secondly, it would not allow grammatical uses of the impoverished items to be differentiated from ungrammatical uses. With an impoverished lexicon, any use (grammatical or not) of "other" or "another" would be flagged as an error, since it would involve using a lexical entry that does not have all of the features that the standard entry has. Since the lexical item would not have the `agr` specification, it could not match the rule that requires agreement between determiners and nouns.

### Implementation

For these reasons, we decided not to use different lexical entries to model the different stages of acquisition. Instead, we use mal-rules, the same mechanism that we are using to model syntactic changes. A standard (grammatical) DP (Determiner Phrase) rule has the following format:

** 12**.  * `DP(agr ?a) `$\rightarrow$` Det(agr ?a) NP(agr ?a)` *

We initially tried simply eliminating the references to agreement between the NP and the determiner, as in the following mal-rule:

** 13**.  * `DP(error +)(agr ?a) `$\rightarrow$` Det NP(agr ?a)` *

This has the advantage of flagging any deviant DPs as having the error feature, since ungrammatical DPs will trigger the mal-rule , but won't trigger . However, a grammatical DP (e.g. "another child") fires both the mal-rule and the grammatical rule . Not only did this behavior cause the parser to slow down very significantly, since it effectively doubled the number of DPs in a sentence, but it also has the potential to report an error when one does not exist. We also briefly considered using impoverishment rules on specific categories. For example, we could have used a rule stating that determiners have all possible agreement values. This has the effect of eliminating agreement as a barrier to unification, much as would be expected if the learner has no knowledge of agreement on determiners. However, this solution has a problem very similar to that of the previous possible solution: all determiners in the input could suddenly have two entries in the chart – one with the actual agreement, one with the impoverished agreement. These would then both be used in parsing, leading to another explosion in the number of parses.

We finally ended up building a set of rules that matches just the ungrammatical possibilities, i.e. they do not allow a grammatical structure to fire both the mal-rule and the normal rule. The present set of rules for determiner-NP agreement include the following:

** 14**.  * `DP(agr ?a) `$\rightarrow$` Det (agr ?a) NP (agr ?a)` *

** 15**.  * `DP(agr s)(error +) `$\rightarrow$` Det(agr (?!a s)) NP(agr s)` *

** 16**.  * `DP(agr p)(error +) `$\rightarrow$` Det(agr (?!a p)) NP(agr p)` *

This solution required using the negation operator "!" present in our parser to specify that a `Det` not allow singular/plural agreement. However, this feature is limited in the present implementation to constant values, i.e. we can't negate a variable. This solution achieves the major goal of not introducing extraneous parses for grammatical constituents. However, it achieves this goal at some cost. Namely, we are forced to increase the number of rules in order to accomplish the task.

### Future plans

We are presently working on the implementation of a variant of unification that will allow us to do the job with fewer rules. The new operation will work in the following sort of rule:

** 17**.  * `DP (agr ?a)`$\rightarrow$` Det(agr ?!a) NP(agr ?a)` *

This rule will be interpreted as follows: the `agr` values between the `DP` and the `NP` will be the same, and none of the values in `Det` will be allowed to be in the agreement values for the `NP` and the `DP`. This will allow the rule to fire precisely when there are no possible ways to unify the values between the `Det` and the `NP`, i.e. none of the `agr` values for the `Det` will be allowed in the variable `?a`. Thus, this rule will only fire for ungrammatical constructions.

# Grammar Coverage/User Interface

The ICICLE grammar is a broad-coverage grammar designed to parse a wide variety of both grammatical sentences and sentences containing errors. It is built around the COMLEX Syntax 2.2 lexicon , which contains approximately 38,000 different syntactic head words. We have a simple set of rules that allows for inflection, thereby doubling the number of noun forms, while giving us three to four times as many verb forms as there are heads. Thus we can handle approximately 40,000 noun forms, 8,000 adjectives, and well over 15,000 verb forms. In addition, unknown words coming into the system are assumed to be proper nouns, thus expanding the number of words handled even further.

The grammar itself contains approximately 25 different adjectival subcategorizations, including subcategorizations requiring an extraposed structure (the "it" in "it is true that he is here"). We also include half a dozen noun complementation types. We have approximately 110 different verb complementation frames, many of which are indexed for several different subcategorizations. The grammar is also able to account for verb-particle constructions when the verb is adjacent to the particle, as well as when they are separated (e.g. "I called him up").

Additionally, the grammar allows for various different types of subjects, including infinitivals with and without subjects ("to fail a class is unfortunate", "for him to fail the class is irresponsible"). It handles yes/no questions, wh-questions, and both subject and object relative clauses.

The grammar has only limited abilities concerning coordination $-$ it only allows limited constituent coordination, and does not allow non-constituent coordination (e.g. "I saw and he hit the ball") at all. It is also fairly weak in its handling of adjunct subordinate clauses. The population we are concerned with also has significant trouble with this, in particular there is a strong propensity towards over-using "because". Adverbs are also problematic, in that the system is not yet able to differentiate what position a given adverb should be able to take in a sentence, thus no errors in adverb placement can be flagged. We are presently in the process of integrating a new version of the lexicon that includes features specifying what each adverb can attach to. Once this is done, we expect to be able to process adverbs quite effectively.

The user interface presently consists of a main window where the user can input the text and control parsing, file access, etc. After parsing, the sentences are highlighted with different colors corresponding to different types of errors. When the user double-clicks on a sentence, a separate "fix-it" window is displayed with the sentence in question, along with descriptions of the errors. The user can click on the errors and the system will highlight the part of the sentence where the error occurred. For example, in the sentence "I see a boys", only "a boys" will be highlighted. The "fix-it" window also allows the user to change the sentence and then re-parse it. If the changes are acceptable to the user, the new sentence can be substituted back into the main text.

# Evaluation of Error Recognition

An evaluation of the grammar was conducted on a variety of sentences pulled from the corpus of ASL natives. The corpus contains essays written by ASL natives which is annotated with references to different types of errors in the sentences. The focus for this paper was on recognition of agreement-type problems, and as such we pulled out all of the sentences that had been marked with the following errors:

-   NUM: Number problems, which are typically errors in subject-verb agreement

-   ED: extra determiner

-   MD: missing determiner for an NP that requires a determiner

-   ID: incorrect determiner

In addition to testing sentences with these problems, we also tested fully grammatical sentences from the same corpus, to see if we could correctly differentiate between grammatical and ungrammatical sentences that might be produced by our target user group.

After gathering the sentences from the database, we cut them down to mono-clausal sentences wherever possible, due to the fact that the handling of adjunct clauses is not yet complete (see §). An example of the type of sentence that had to be divided is the following:

** 18**.  * They should communicate each other because the communication is very important to understand each other. *

This sentence was divided into "They should communicate each other" and "the communication is very important to understand each other." In addition to separating the clauses, we also fixed the spelling errors in the sentences to be tested since spelling correction is beyond the scope of the current implementation.

## Results for Ungrammatical Sentences

We ended up with 79 sentences to test for the determiner and agreement errors. Of these 79 sentences, 44 (56%) parse with the expected type of error. Another 23 (29%) have no parses that cover the entire sentence, and 12 (15%) parse as having no errors at all.

A number of the sentences that had been flagged with errors in the database were actually grammatical sentences, but were deemed inappropriate in context. Thus, sentences like the following were tagged with errors in the corpus:

** 19**.  * I started to attend the class last Saturday. *

It was evident from the context that this sentence should have had "classes" rather than "the class." Of the 12 sentences that were parsed as error-free, five were actually syntactically and semantically acceptable, but were inappropriate for their contexts, as in the previous example. Another four had pragmatic/semantic problems, but were syntactically well-formed, as in

** 20**.  * I want to succeed in jobs anywhere. *

Thus, there are really only three sentences that do not have a parse with the appropriate error. Since this parser is a syntactic parser, it should not be expected to find the semantic/pragmatic errors, nor should it know if the sentence was inappropriate for its context in the essay. If we eliminate the nine sentences that are actually grammatical in isolation, we are left with 70 sentences, of which 44 (63%) have parses with the expected error, three (4%) are wrongly accepted as grammatical, and 23 (33%) do not parse.

In terms of evaluating these results for the purposes of the system, we must consider the implications of the various categories. 63% would trigger tutoring, and 33% would be tagged as problematic, but would have no information about the type of error. In only 4% of sentences containing errors would the system incorrectly indicate that no errors are present.

## Results for Grammatical Sentences

We also tested the system on 101 grammatical sentences that were pulled from the same corpus. These sentences were modified in the same way as the ungrammatical ones, with multi-clausal sentences being divided up into mono-clausal sentences. Of these 101 sentences, 89 (88%) parsed as having no errors, 3 (3%) parsed with errors, and the remaining 8 (8%) did not parse.

The present implementation of the grammar suffers from poor recognition of coordination, even within single clauses. Five of the eleven sentences that did not return an error-free parse suffered from this limitation. We expect to be able to improve the numbers significantly by including in the grammar some recognition of punctuation, which, due to technical problems, is presently filtered out of the input before the parser has a chance to use it.

# Conclusions and Future Work

Future work will include extending the grammar to better deal with coordination and adjunct clauses. We will also continue to work on the negation operator and the propagation of the ` missing` feature discussed above. In order to cut down on the number of parses, as well as to make it easier to decide which is the appropriate parse to correct, we have recently switched to a best-first parsing strategy. This should allow us to model which rules are most likely to be used by a given user, with the mal-rules corresponding to the constructions currently being acquired having a higher probability than those that the learner has already mastered. However, at the moment we have simply lowered the probabilities of all mal-rules, so that any grammatical parses are generated first, followed by the "ungrammatical" parses.

As we have shown, this system does a good job of flagging ungrammatical sentences produced by the target population, with a high proportion of the flagged sentences containing significant information about the type and location of the error. Our continuing work will hopefully improve these percentages, and couple this recognition component with an intelligent tutoring phase.

[^1]: This work was supported by NSF Grant \#SRS9416916.

[^2]: While our analysis so far has only indicated that determiner omissions have this property, we do not want to rule out the possibility that other combinations of omission errors might be found to occur as well.

[^3]: "Another" and "other" are not separate lexical items in ASL.