abstract: In this work we present a framework for the recognition of natural scene text. Our framework does not require any human-labelled data, and performs word recognition on the whole image holistically, departing from the character based recognition systems of the past. The deep neural network models at the centre of this framework are trained solely on data produced by a synthetic text generation engine – synthetic data that is highly realistic and sufficient to replace real data, giving us infinite amounts of training data. This excess of data exposes new possibilities for word recognition models, and here we consider three models, each one "reading" words in a different way: via 90k-way dictionary encoding, character sequence encoding, and bag-of-N-grams encoding. In the scenarios of language based and completely unconstrained text recognition we greatly improve upon state-of-the-art performance on standard datasets, using our fast, simple machinery and requiring zero data-acquisition costs.
author: Max Jaderberg
Karen Simonyan
Andrea Vedaldi
Andrew Zisserman
bibliography: bib/shortstrings.bib; bib/vgg_local.bib; bib/vgg_other.bib; bib/current.bib
title: Synthetic Data and Artificial Neural Networks for
  Natural Scene Text Recognition

Visual Geometry Group, University of Oxford
`{max,karen,vedaldi,email@example.com`

# Introduction

Text recognition in natural images, *scene text recognition*, is a challenging but wildly useful task. Text is one of the basic tools for preserving and communicating information, and a large part of the modern world is designed to be interpreted through the use of labels and other textual cues. This makes scene text recognition imperative for many areas in information retrieval, in addition to being crucial for human-machine interaction.

While the recognition of text within scanned documents is well studied and there are many document OCR systems that perform very well, these methods do not translate to the highly variable domain of scene text recognition. When applied to natural scene images, traditional OCR techniques fail as they are tuned to the largely black-and-white, line-based environment of printed documents, while text occurring in natural scene images suffers from inconsistent lighting conditions, variable fonts, orientations, background noise, and imaging distortions.

To effectively recognise scene text, there are generally two stages: word detection and word recognition. The detection stage generates a large set of word bounding box candidates, and is tuned for speed and high recall. Previous work uses sliding window methods  or region grouping methods  very successfully for this. Subsequently, these candidate detections are recognised, and this recognition process allows for filtering of false positive word detections. Recognition is therefore a far more challenging problem and it is the focus of this paper.

While most approaches recognize individual characters by pooling evidence locally, Goodfellow *et al.*  do so from the image of the whole character string using a convolutional neural network (CNN) . They apply this to street numbers and synthetic CAPTCHA recognition obtaining excellent results. Inspired by this approach, we move further in the direction of holistic word classification for scene text, and make two important contributions. Firstly, we propose a state-of-the-art CNN text recogniser that also pools evidence from images of entire words. Crucially, however, we regress all the characters simultaneously, formulating this as a classification problem in a large lexicon of 90k possible words (Sect. ). In order to do so, we show how CNNs can be efficiently trained to recognise a very large number of words using *incremental training*. While our lexicon is restricted, it is so large that this hardly constitutes a practical limitation. Secondly, we show that this state-of-the-art recogniser can be trained *purely from synthetic data*. This result is highly non-trivial as, differently from CAPTCHA, the classifier is then applied to real images. While synthetic data was used previously for OCR, it is remarkable that this can be done for scene text, which is significantly less constrained. This allows our framework to be seamlessly extended to larger vocabularies and other languages without any human-labelling cost. In addition to these two key contributions, we study two alternative models – a character sequence encoding model with a modified formulation to that of  (Sect. ), and a novel bag-of-N-grams encoding model which predicts the unordered set of N-grams contained in the word image (Sect. ).

A discussion of related work follows immediately and our data generation system described after in Sect. . Our deep learning word recognition architectures are presented in Sect. , evaluated in Sect. , and conclusions are drawn in Sect. .

Traditional text recognition methods are based on sequential character classification by either sliding windows  or connected components , after which a word prediction is made by grouping character classifier predictions in a left-to-right manner. The sliding window classifiers include random ferns  in Wang *et al.* , and CNNs in . Both  and  use a small fixed lexicon as a language model to constrain word recognition.

More recent works such as  make use of over-segmentation methods, guided by a supervised classifier, to generate candidate proposals which are subsequently classified as characters or false positives. For example, PhotoOCR  uses binarization and a sliding window classifier to generate candidate character regions, with words recognised through a beam search driven by classifier scores followed by a re-ranking using a dictionary of 100k words.  uses the convolutional nature of CNNs to generate response maps for characters and bigrams which are integrated to score lexicon words.

In contrast to these approaches based on character classification, the work by  instead uses the notion of holistic word recognition.  still rely on explicit character classifiers, but construct a graph to infer the word, pooling together the full word evidence. Rodriguez *et al.*  use aggregated Fisher Vectors  and a Structured SVM framework to create a joint word-image and text embedding.  use whole word-image features to recognize words by comparing to simple black-and-white font-renderings of lexicon words.

Goodfellow *et al.*  had great success using a CNN with multiple position-sensitive character classifier outputs (closely related to the character sequence model in Sect. ) to perform street number recognition. This model was extended to CAPTCHA sequences (up to 8 characters long) where they demonstrated impressive performance using synthetic training data for a synthetic problem (where the generative model is known), but we show that synthetic training data can be used for a real-world data problem (where the generative model is unknown).

# Synthetic Data Engine

This section describes our scene text rendering algorithm. As our CNN models take whole word images as input instead of individual character images, it is essential to have access to a training dataset of cropped word images that covers the whole language or at least a target lexicon. While there are some publicly available datasets from ICDAR , the Street View Text (SVT) dataset  and others, the number of full word image samples is only in the thousands, and the vocabulary is very limited. These limitations have been mitigated before by mining for data or having access to large proprietary datasets , but neither of these approaches are wholly accessible or scalable.

Here we follow the success of some synthetic character datasets  and create a synthetic word data generator, capable of emulating the distribution of scene text images. This is a reasonable goal, considering that much of the text found in natural scenes is computer-generated and only the physical rendering process (*e.g.* printing, painting) and the imaging process (*e.g.* camera, viewpoint, illumination, clutter) are not controlled by a computer algorithm.

Fig.  illustrates the generative process and some resulting synthetic data samples. These samples are composed of three separate image-layers – a background image-layer, foreground image-layer, and optional border/shadow image-layer – which are in the form of an image with an alpha channel. The synthetic data generation process is as follows:

1.  *Font rendering* – a font is randomly selected from a catalogue of over 1400 fonts downloaded from Google Fonts. The kerning, weight, underline, and other properties are varied randomly from arbitrarily defined distributions. The word is rendered on to the foreground image-layer's alpha channel with either a horizontal bottom text line or following a random curve.

2.  *Border/shadow rendering* – an inset border, outset border or shadow with a random width may be rendered from the foreground.

3.  *Base coloring* – each of the three image-layers are filled with a different uniform color sampled from clusters over natural images. The clusters are formed by k-means clustering the three color components of each image of the training datasets of  into three clusters.

4.  *Projective distortion* – the foreground and border/shadow image-layers are distorted with a random, full-projective transformation, simulating the 3D world.

5.  *Natural data blending* – each of the image-layers are blended with a randomly-sampled crop of an image from the training datasets of ICDAR 2003 and SVT. The amount of blend and alpha blend mode (*e.g.* normal, add, multiply, burn, max, *etc.*) is dictated by a random process, and this creates an eclectic range of textures and compositions. The three image-layers are also blended together in a random manner, to give a single output image.

6.  *Noise* – Gaussian noise, blur, and JPEG compression artefacts are introduced to the image.

The word samples are generated with a fixed height of 32 pixels, but with a variable width. Since the input to our CNNs is a fixed-size image, the generated word images are rescaled so that the width equals 100 pixels. Although this does not preserve the aspect ratio, the horizontal frequency distortion of image features most likely provides the word-length cues. We also experimented with different padding regimes to preserve the aspect ratio, but found that the results are not quite as good as with resizing.

The synthetic data is used in place of real-world data, and the labels are generated from a corpus or dictionary as desired. By creating training datasets much larger than what has been used before, we are able to use data-hungry deep learning algorithms to train richer, whole-word-based models.

# Models

In this section we describe three models for visual recognition of scene text words. All use the same framework of generating synthetic text data (Sect. ) to train deep convolutional networks on whole-word image samples, but with different objectives, which correspond to different methods of reading. Sect.  describes a model performing pure word classification to a large dictionary, explicitly modelling the entire known language. Sect.  describes a model that encodes the character at each position in the word, making no language assumptions to naively predict the sequence of characters in an image. Sect.  describes a model that encodes a word as a bag-of-N-grams, giving a compositional model of words as not only a collection of characters, but of 2-grams, 3-grams, and more generally, N-grams.

## Encoding Words

This section describes our first model for word recognition, where words $w$ are constrained to be selected in a pre-defined dictionary $\mathcal{W}$. We formulate this as multi-class classification problem, with one class per word. While the dictionary $\mathcal{W}$ of a natural language may seem too large for this approach to be feasible, in practice an advanced English vocabulary, including different word forms, contains only around 90k words, which is large but manageable.

In detail, we propose to use a CNN classifier where each word $w\in\mathcal{W}$ in the lexicon corresponds to an output neuron. We use a CNN with four convolutional layers and two fully connected layers. Rectified linear units are used throughout after each weight layer except for the last one. In forward order, the convolutional layers have 64, 128, 256, and 512 square filters with an edge size of 5, 5, 3, and 3. Convolutions are performed with stride 1 and there is input feature map padding to preserve spatial dimensionality. $2 \times 2$ max-pooling follows the first, second and third convolutional layers. The fully connected layer has 4096 units, and feeds data to the final fully connected layer which performs classification, so has the same number of units as the size of the dictionary we wish to recognize. The predicted word recognition result $w^*$ out of the set of all dictionary words $\mathcal{W}$ in a language $\mathcal{L}$ for a given input image $x$ is given by $w^* = \arg\max_{w \in \mathcal{W}} P(w|x,\mathcal{L})$. Since $P(w|x,\mathcal{L}) = \frac{P(w|x)P(w|\mathcal{L})P(x)}{P(x|\mathcal{L})P(w)}$ and with the assumptions that $x$ is independent of $\mathcal{L}$ and that prior to any knowledge of our language all words are equally probable, our scoring function reduces to $w^* = \arg\max_{w \in \mathcal{W}} P(w|x)P(w|\mathcal{L})$. The per-word output probability $P(w|x)$ is modelled by the softmax scaling of the final fully connected layer, and the language based word prior $P(w|\mathcal{L})$ can be modelled by a lexicon or frequency counts. A schematic of the network is shown in Fig.  (a).

We train the network by back-propagating the standard multinomial logistic regression loss with dropout , which improves generalization. Optimization uses stochastic gradient descent (SGD), dynamically lowering the learning rate as training progresses. With uniform sampling of classes in training data, we found the SGD batch size must be at least a fifth of the total number of classes in order for the network to train.

For very large numbers of classes (*i.e.* over 5k classes), the SGD batch size required to train effectively becomes large, slowing down training a lot. Therefore, for large dictionaries, we perform *incremental training* to avoid requiring a prohibitively large batch size. This involves initially training the network with 5k classes until partial convergence, after which an extra 5k classes are added. The original weights are copied for the original 5k classes, with the new classification layer weights being randomly initialized. The network is then allowed to continue training, with the extra randomly initialized weights and classes causing a spike in training error, which is quickly trained away. This process of allowing partial convergence on a subset of the classes, before adding in more classes, is repeated until the full number of desired classes is reached. In practice for this network, the CNN trained well with initial increments of 5k classes, and after 20k classes is reached the number of classes added at each increment is increased to 10k.

## Encoding Sequences of Characters

This section describes a different model for word recognition. Rather than having a single large dictionary classifier as in Sect. , this model uses a single CNN with multiple independent classifiers, each one predicting the character at each position in the word. This character sequence encoding model is a complete departure from the dictionary-constrained model, as this allows entirely unconstrained recognition of words.

A word $w$ of length $N$ is modelled as a sequence of characters such that $w = (c_1, c_2, \dots, c_N)$ where each $c_i \in \mathcal{C}= \{1,2,\dots,36\}$ represents a character at position $i$ in the word, from the set of 10 digits and 26 letters. Each $c_i$ can be predicted with a single classifier, one for each character in the word. However, since words have variable length $N$ which is unknown at test time, we fix the number of characters to 23, the maximum length of a word in the training set, and introduce a null character class. Therefore a word is represented by a string $w = (\mathcal{C}\cup \{\phi\})^{23}$. Then for a given image $x$, each character is predicted as $c_i^* = \arg\max_{c_i \in \mathcal{C}\cup \{\phi\}} P(c_i|\Phi(x))$. $P(c_i|\Phi(x))$ is given by the $i$-th classifier acting on a single set of shared CNN features $\Phi(x)$.

The base CNN has the same structure as the first five layers of Sect. : four convolutional layers followed by a fully connected layer, giving $\Phi(x)$. The output of the fully connected layer is then fed to 23 separate fully connected layers with 37 neurons each, one for each character class. These fully connected layers are independently softmax normalized and can be interpreted as the probabilities $P(c_i|\Phi(x))$ of the width-resized input image $x$. Fig.  (b) illustrates this model. The model is trained as in Sect.  on purely synthetic data by SGD with dropout regularisation, back-propagating gradients from each 23 softmax classifier to the base net.

This sequential character encoding model is similar to the model used by Goodfellow *et al.* in . Although the model of  is not applied to scene text (only street numbers and CAPTCHA puzzles), it uses a separate character classifier for each letter in the word, able to recognise numbers up to 5 digits long and CAPTCHAs up to 8 characters long. However, rather than incorporating a no-character class in each character positions's classifier, a further length classifier is trained to output the predicted length of the word. This requires a final post-processing stage to find the optimal word prediction given the character classifier outputs and the length classifier output. We achieve a similar effect but without requiring any post processing – the word can be read directly from the CNN output, stripping the no-character class predictions.

## Encoding Bags of N-grams

This section describes our last word recognition model, which exploits compositionality to represent words. In contrast to the sequential character encoding of Sect. , words can be seen as a composition of an unordered set of character N-grams, a *bag-of-N-grams*. In the following, if $s\in \mathcal{C}^N$ and $w\in\mathcal{C}^M$ are two strings, the symbol $s \subset w$ indicates that $s$ is a substring of $w$. An $N$-gram of word $w$ is a substring $s \subset w$ of length $|w|=N$. We will denote with $G_N(w) = \{ s : s\subset w \,\wedge\,|s|\leq N\}$ the set of all grams of word $w$ of length up to $N$ and with $G_N
=\cup_{w\in\mathcal{W}} G_N(w)$ the set of all such grams in the language. For example, $G_3(\text{spires})=\{\text{s},\text{p},\text{i},\text{r},\text{e},\text{s},\text{sp},\text{pi},\text{ir},\text{re},\text{es},\text{spi},\text{pir},\text{ire},\text{res}\}$. This method of encoding variable length sequences is similar to *Wickelphone* phoneme-encoding methods .

Even for small values of $N$, $G_N(w)$ encodes each word $w\in\mathcal{W}$ nearly uniquely. For example, with $N=4$, this map has only 7 collisions out of a dictionary of 90k words. The encoding $G_N(w)$ can be represented as a $|G_N|$-dimensional binary vector of gram occurrences. This vector is very sparse, as on average $|G_N(w)|\approx 22$ whereas $|G_N|=10k$. Given $w$, we predict this vector using the same base CNN as in Sect.  and Sect. , but now have a final fully connected layer with $|G_N|$ neurons to represent the encoding vector. The scores from the fully connected layer can be interpreted as probabilities of an N-gram being present in the image by applying the logistic function to each neuron. The CNN is therefore learning to recognise the presence of each N-gram somewhere within the input image.

With a logistic function, the training problem becomes that of $|G_N|$ separate binary classification tasks, and so we back-propagate the logistic regression loss with respect to each N-gram class independently. To jointly train a whole range of N-grams, some of which occur very frequently and some barely at all, we have to scale the gradients for each N-gram class by the inverse frequency of their appearance in the training word corpus. We also experimented with hinge loss and simple regression to train but found frequency weighted binary logistic regression was superior. As with the other models, we use dropout and SGD.

# Evaluation

This section evaluates our three text recognition models. Sect.  describes the benchmark data, Sect.  the implementation details, and Sect.  the results of our methods, that improve on the state of the art.

## Datasets

A number of standard datasets are used for the evaluation of our systems – ICDAR 2003, ICDAR 2013, Street View Text, and IIIT5k. **ICDAR 2003**  is a scene text recognition dataset, with the test set containing 251 full scene images and 860 groundtruth cropped images of the words contained with the full images. We follow the standard evaluation protocol by  and perform recognition on only the words containing only alphanumeric characters and at least three characters. The test set of 860 cropped word images is referred to as IC03. The lexicon of all test words is IC03-Full, and the per-image 50 word lexicons defined by  and used in  are referred to as IC03-50. There is also the lexicon of all groundtruth test words – IC03-Full which contains 563 words. **ICDAR 2013**  test dataset contains 1015 groundtruth cropped word images from scene text. Much of the data is inherited from the ICDAR 2003 datasets. We refer to the 1015 groundtruth cropped words as IC13. **Street View Text**  is a more challenging scene text dataset than the ICDAR datasets. It contains 250 full scene test images downloaded from Google Street View. The test set of 647 groundtruth cropped word images is referred to as SVT. The lexicon of all test words is SVT-Full (4282 words), and the smaller per-image 50 word lexicons defined by  and used in  are referred to as SVT-50. **IIIT 5k-word**  test dataset contains 3000 cropped word images of scene text downloaded from Google image search. Each image has an associated 50 word lexicon (IIIT5k-50) and 1k word lexicon (IIIT5k-1k).

For training, validation and large-lexicon testing we generate datasets using the synthetic text engine from Sect. . 4 million word samples are generated for the IC03-Full and SVT-Full lexicons each, referred to as Synth-IC03 and Synth-SVT respectively. In addition, we use the dictionary from Hunspell, a popular open source spell checking system, combined with the ICDAR and SVT test words as a 50k word lexicon. The 50k Hunspell dictionary can also be expanded to include different word endings and combinations to give a 90k lexicon. We generate 9 million images for the 50k word lexicon and 9 million images for the 90k word lexicon. The 9 million image synthetic dataset covering 90k words, Synth, is available for download at <http://www.robots.ox.ac.uk/~vgg/data/text/>.

## Implementation Details

We perform experiments on all three encoding models described in Sect. . We will refer to the three models as DICT, CHAR, and NGRAM for the dictionary encoding model, character sequence encoding model, and N-gram encoding model respectively. The input images to the CNNs are greyscale and resized to $32 \times 100$ without aspect ratio preservation. The only preprocessing, performed on each sample individually, is the sample mean subtraction and standard deviation normalization (after resizing), as this was found to slightly improve performance. Learning uses a custom version of `Caffe` .

All CNN training is performed solely on the Synth training datasets, with model validation performed on a 10% held out portion. The number of character classifiers in the CHAR character sequence encoding models is set to 23 (the length of the largest word in our 90k dictionary). In the NGRAM models, the number of N-grams in the N-gram classification dictionary is set to 10k. The N-grams themselves are selected as the N-grams with at least 10 appearances in the 90k word corpus – this equates to 36 1-grams (the characters), 522 2-grams, 3965 3-grams, and 5477 4-grams, totalling 10k.

In addition to the CNN model defined in Sect. , we also define larger CNN, referred to as DICT+2, CHAR+2, and NGRAM+2. The larger CNN has an extra $3 \times 3$ convolutional layer with 512 filters before the final pooling layer, and an extra 4096 unit fully connected layer after the original 4096 unit fully connected layer. Both extra layers use rectified linear non-linearities. Therefore, the total structure for the DICT+2 model is conv-pool-conv-pool-conv-conv-pool-conv-fc-fc-fc, where conv is a convolutional layer, pool is a max-pooling layer and fc is a fully connected layer. We train these larger models to investigate the effect of additional model capacity, as the lack of over-fitting experienced on the basic models is suspected to indicate under-capacity of the models.

## Experiments

<table>
<caption><em>Left:</em> The word recognition accuracy for the different proposed models with different trained lexicons. Where a lexicon is not specified for a dataset, the only language constraints are those imposed by the model itself. The fixed lexicon CHAR model results (IC03-50 and SVT-50) are obtained by selecting the lexicon word with the minimum edit distance to the predicted character sequence. <em>Right:</em> Some random example results from the SVT and ICDAR 2013 dataset. D denotes DICT+2-90k with no lexicon, D-50 the DICT+2-90k model constrained to the image's 50 word lexicon, C denotes the CHAR+2 model with completely unconstrained recognition, and C-50 gives the result of the closest edit distance 50-lexicon word.</caption>
<thead>
<tr>
<th style="text-align: center;">Model</th>
<th style="text-align: left;">Trained Lexicon</th>
<th style="text-align: center;">Synth</th>
<th style="text-align: center;">IC03-50</th>
<th style="text-align: center;">IC03</th>
<th style="text-align: center;">SVT-50</th>
<th style="text-align: center;">SVT</th>
<th style="text-align: center;">IC13</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;">DICT-IC03-Full</td>
<td style="text-align: left;">IC03-Full</td>
<td style="text-align: center;">98.7</td>
<td style="text-align: center;">99.2</td>
<td style="text-align: center;">98.1</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
</tr>
<tr>
<td style="text-align: left;"><p>DICT-SVT-Full</p></td>
<td style="text-align: left;">SVT-Full</td>
<td style="text-align: center;">98.7</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">96.1</td>
<td style="text-align: center;">87.0</td>
<td style="text-align: center;">-</td>
</tr>
<tr>
<td style="text-align: left;">DICT-50k</td>
<td style="text-align: left;">50k</td>
<td style="text-align: center;">93.6</td>
<td style="text-align: center;">99.1</td>
<td style="text-align: center;">92.1</td>
<td style="text-align: center;">93.5</td>
<td style="text-align: center;">78.5</td>
<td style="text-align: center;">92.0</td>
</tr>
<tr>
<td style="text-align: left;"><p>DICT-90k</p></td>
<td style="text-align: left;">90k</td>
<td style="text-align: center;">90.3</td>
<td style="text-align: center;">98.4</td>
<td style="text-align: center;">90.0</td>
<td style="text-align: center;">93.7</td>
<td style="text-align: center;">73.0</td>
<td style="text-align: center;">86.3</td>
</tr>
<tr>
<td style="text-align: left;">DICT+2-90k</td>
<td style="text-align: left;">90k</td>
<td style="text-align: center;">95.2</td>
<td style="text-align: center;">98.7</td>
<td style="text-align: center;">93.1</td>
<td style="text-align: center;">95.4</td>
<td style="text-align: center;">80.7</td>
<td style="text-align: center;">90.8</td>
</tr>
<tr>
<td style="text-align: left;"><p>CHAR</p></td>
<td style="text-align: left;">90k</td>
<td style="text-align: center;">71.0</td>
<td style="text-align: center;">94.2</td>
<td style="text-align: center;">77.0</td>
<td style="text-align: center;">87.8</td>
<td style="text-align: center;">56.4</td>
<td style="text-align: center;">68.8</td>
</tr>
<tr>
<td style="text-align: left;">CHAR+2</td>
<td style="text-align: left;">90k</td>
<td style="text-align: center;">86.2</td>
<td style="text-align: center;">96.7</td>
<td style="text-align: center;">86.2</td>
<td style="text-align: center;">92.6</td>
<td style="text-align: center;">68.0</td>
<td style="text-align: center;">79.5</td>
</tr>
<tr>
<td style="text-align: left;"><p>NGRAM-NN</p></td>
<td style="text-align: left;">90k</td>
<td style="text-align: center;">25.1</td>
<td style="text-align: center;">92.2</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">84.5</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
</tr>
<tr>
<td style="text-align: left;">NGRAM+2-NN</td>
<td style="text-align: left;">90k</td>
<td style="text-align: center;">27.9</td>
<td style="text-align: center;">94.2</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">86.6</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
</tr>
</tbody>
</table>

We evaluate each of our three models on challenging text recognition benchmarks. First, we measure the accuracy on a large dataset, containing the images of words from the full lexicon (up to 90k words depending on the model). Due to the lack of human-annotated natural image datasets of such scale, we use the test split of our Synth dataset (Sect. ). This allows us to assess how well our models can discriminate between a large number of words. Second, we consider the standard benchmarks IC03 , SVT , and IC13 , which contain natural scene images, but cover smaller word lexicons. The evaluation on these datasets allows for a fair comparison against the state of the art. The results are shown in Table  and Table .

For the DICT model, we train a model with only the words from the IC03-Full lexicon (DICT-IC03-Full), a model with only the words from the SVT-Full lexicon (DICT-SVT-Full), as well as models for the 50k and 90k lexicons – DICT-50k, DICT-90k, and DICT+2-90k. When a small lexicon is provided, we set the language prior $P(w|\mathcal{L})$ to be equal probability for lexicon words, otherwise zero. In the absence of a small lexicon, $P(w|\mathcal{L})$ is simply the frequency of word $w$ in a corpus (we use the opensubtitles.org English corpus) normalized according to the power law.

The results in Table  show exceptional performance for the dictionary based models. When the model is trained purely for a dataset's corpus of words (DICT-IC03-Full and DICT-SVT-Full), the 50-lexicon recognition problem is largely solved for both ICDAR 2003 and SVT, achieving 99.2% and 96.1% word recognition accuracy respectively, that is 7 mistakes out of 860 in the ICDAR 2003 test set, of which most are completely illegible. The Synth dataset performs very closely to that of the ICDAR 2003 dataset, confirming that the synthetic data is close to the real world data.

Drastically increasing the size of the dictionary to 50k and 90k words gives very little degradation in 50-lexicon accuracy. However without the 50-lexicon constraint, as expected the 50k and 90k dictionary models perform significantly worse than when the dictionary is constrained to only the groundtruth words – on SVT, the word classification from only the 4282 groundtruth word set yields 87% accuracy, whereas increasing the dictionary to 50k reduces the accuracy to 78.5%, and the accuracy is further reduced to 73.0% with 90k word classes. Incorporating the extra layers in to the network with DICT+2-90k increases the accuracy a lot, giving 80.7% on SVT for full 90k-way classification, almost identical to a dictionary of 50k with the basic CNN architecture.

We also investigate the contribution that the various stages of the synthetic data generation engine make to real-world recognition accuracy. Figure  (*left*) shows DICT-IC03-Full and DICT-SVT-Full accuracy when trained identically but with different levels of sophistication of synthetic training data. As more sophisticated training data is used, the recognition accuracy increases – the addition of random image-layer colouring causing a significant increase in performance (+44% on IC03 and +40% on SVT), as does the addition of natural image blending (+1% on IC03 and +6% on SVT).

<table>
<caption>Comparison to previous methods. The ICDAR 2013 results given are case-insensitive. Bolded results outperform previous state-of-the-art methods. The baseline method is from a commercially available OCR system. </caption>
<thead>
<tr>
<th style="text-align: center;">Model</th>
<th style="text-align: center;">IC03-50</th>
<th style="text-align: center;">IC03-Full</th>
<th style="text-align: center;">IC03-50k</th>
<th style="text-align: center;">SVT-50</th>
<th style="text-align: center;">SVT</th>
<th style="text-align: center;">IC13</th>
<th style="text-align: center;">IIIT5k-50</th>
<th style="text-align: center;">IIIT5k-1k</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;"><p><em>Baseline ABBYY</em> <span class="citation" data-cites="Wang11"></span></p></td>
<td style="text-align: center;">56.0</td>
<td style="text-align: center;">55.0</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">35.0</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">24.3</td>
<td style="text-align: center;">-</td>
</tr>
<tr>
<td style="text-align: left;">Wang <span class="citation" data-cites="Wang11"></span></td>
<td style="text-align: center;">76.0</td>
<td style="text-align: center;">62.0</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">57.0</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
</tr>
<tr>
<td style="text-align: left;"><p>Mishra <span class="citation" data-cites="Mishra12"></span></p></td>
<td style="text-align: center;">81.8</td>
<td style="text-align: center;">67.8</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">73.2</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
</tr>
<tr>
<td style="text-align: left;">Novikova <span class="citation" data-cites="Novikova12"></span></td>
<td style="text-align: center;">82.8</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">72.9</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">64.1</td>
<td style="text-align: center;">57.5</td>
</tr>
<tr>
<td style="text-align: left;"><p>Wang & Wu <span class="citation" data-cites="Wang12"></span></p></td>
<td style="text-align: center;">90.0</td>
<td style="text-align: center;">84.0</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">70.0</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
</tr>
<tr>
<td style="text-align: left;">Goel <span class="citation" data-cites="Goel13"></span></td>
<td style="text-align: center;">89.7</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">77.3</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
</tr>
<tr>
<td style="text-align: left;"><p>PhotoOCR <span class="citation" data-cites="Bissacco13"></span></p></td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">90.4</td>
<td style="text-align: center;">78.0</td>
<td style="text-align: center;">87.6</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
</tr>
<tr>
<td style="text-align: left;">Alsharif <span class="citation" data-cites="Alsharif13"></span></td>
<td style="text-align: center;">93.1</td>
<td style="text-align: center;">88.6</td>
<td style="text-align: center;">85.1</td>
<td style="text-align: center;">74.3</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
</tr>
<tr>
<td style="text-align: left;"><p>Almazan <span class="citation" data-cites="Almazan14"></span></p></td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">89.2</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">91.2</td>
<td style="text-align: center;">82.1</td>
</tr>
<tr>
<td style="text-align: left;">Yao <span class="citation" data-cites="Yao14"></span></td>
<td style="text-align: center;">88.5</td>
<td style="text-align: center;">80.3</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">75.9</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">80.2</td>
<td style="text-align: center;">69.3</td>
</tr>
<tr>
<td style="text-align: left;"><p>Jaderberg <span class="citation" data-cites="Jaderberg14a"></span></p></td>
<td style="text-align: center;">96.2</td>
<td style="text-align: center;">91.5</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">86.1</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
</tr>
<tr>
<td style="text-align: left;">Gordo <span class="citation" data-cites="Gordo14"></span></td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">90.7</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">93.3</td>
<td style="text-align: center;">86.6</td>
</tr>
<tr>
<td style="text-align: left;"><p>DICT-IC03-Full</p></td>
<td style="text-align: center;"><strong>99.2</strong></td>
<td style="text-align: center;"><strong>98.1</strong></td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
</tr>
<tr>
<td style="text-align: left;">DICT-SVT-Full</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;"><strong>96.1</strong></td>
<td style="text-align: center;"><strong>87.0</strong></td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
</tr>
<tr>
<td style="text-align: left;"><p>DICT+2-90k</p></td>
<td style="text-align: center;"><strong>98.7</strong></td>
<td style="text-align: center;"><strong>98.6</strong></td>
<td style="text-align: center;"><strong>93.3</strong></td>
<td style="text-align: center;"><strong>95.4</strong></td>
<td style="text-align: center;"><strong>80.7</strong></td>
<td style="text-align: center;"><strong>90.8</strong></td>
<td style="text-align: center;"><strong>97.1</strong></td>
<td style="text-align: center;"><strong>92.7</strong></td>
</tr>
<tr>
<td style="text-align: left;">CHAR+2</td>
<td style="text-align: center;"><strong>96.7</strong></td>
<td style="text-align: center;"><strong>94.0</strong></td>
<td style="text-align: center;"><strong>89.5</strong></td>
<td style="text-align: center;"><strong>92.6</strong></td>
<td style="text-align: center;">68.0</td>
<td style="text-align: center;">79.5</td>
<td style="text-align: center;"><strong>95.5</strong></td>
<td style="text-align: center;">85.4</td>
</tr>
<tr>
<td style="text-align: left;"><p>NGRAM+2-SVM</p></td>
<td style="text-align: center;"><strong>96.5</strong></td>
<td style="text-align: center;"><strong>94.0</strong></td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
<td style="text-align: center;">-</td>
</tr>
</tbody>
</table>

The CHAR models are trained for character sequence encoding. The models are trained on image samples of words uniformly sampled from the 90k dictionary.

The output of the model are character predictions for a possible 23 characters of the test image's word. We take the predicted word as the MAP-optimal sequence of characters, stripping any no-character classifications. The constrained lexicon results for IC03-50, IC03-Full, and SVT-50, are obtained by finding the lexicon word with the minimum edit distance to a raw predicted character sequence. Given this is a completely unconstrained recognition, with no language model at all, the results are surprisingly good. The 50-lexicon results are very competitive compared to the other encoding methods. However, we can see the lack of language constraints cause the out-of-lexicon results to be lacklustre, achieving an accuracy of only 79.5% with the CHAR+2 model on ICDAR 2013 as opposed to 90.8% with the DICT+2-90k model. As with the DICT models, increasing the number of layers in the network increases the word recognition accuracy by between 6-8%.

Some example word recognition results with dictionary and character sequence encodings are shown to the right of Table .

The NGRAM model's output is thresholded to result in a binary activation vector of the presence of any of 10k N-grams in a test word. Decoding the N-gram activations into a word could take advantage of a statistical model of the language. Instead, we simply search for the word in the lexicon with the nearest (in terms of the Euclidean distance) N-gram encoding, denoted as NGRAM-NN and NGRAM+2-NN models. This extremely naive method still gives competitive performance, illustrating the discriminative nature of N-grams for word recognition. Instead, one could learn a linear SVM mapping from N-gram encoding to dictionary words, allowing for scalable word recognition through an inverted index of these mappings. We experimented briefly with this on the IC03-Full lexicon – training an SVM for each lexicon word from a training set of Synth data, denoted as NGRAM+2-SVM – and achieve 97% accuracy on IC03-50 and 94% accuracy on IC03-Full. Figure  (*right*) shows the N-gram recognition results for the NGRAM+2 model, thresholded at 0.99 probability.

Table  compares our models to previous work, showing that all three models achieve state-of-the-art results in different lexicon scenarios. With tightly constrained language models such as in DICT-IC03-Full and DICT-SVT-Full, we improve accuracy by +6%. However, even when the models are expanded to be mostly unconstrained, such as with DICT+2-90k, CHAR+2 and NGRAM+2-SVM, our models still outperform previous methods. Considering a complete absence of a language model, the no-lexicon recognition results for the CHAR+2 model on SVT and IC13 are competitive with the system of , and as soon as a language model is introduced in the form of a lexicon for SVT-50, the simple CHAR+2 model gives +2.2% accuracy over . Performance could be further improved by techniques such as model averaging and test-sample augmentation, albeit at a significantly increased computational cost. Our largest model, the DICT+2-90k model comprised of over 490 million parameters, can process a word in 2.2ms on a single commodity GPU.

Our models set a new benchmark for scene text recognition. In a real-world system, the large DICT+2-90k model should be used for the majority of recognition scenarios unless completely unconstrained recognition is required where the CHAR+2 model can be used. However, when looking at the average edit distance of erroneous recognitions, the CHAR+2 model greatly outperforms the DICT+2-90k model, with an average error edit distance of 1.9 compared to 2.5 on IC13, suggesting the CHAR+2 model may be more suitable for a retrieval style application in conjunction with a fuzzy search.

# Conclusion

In this paper we introduced a new framework for scalable, state-of-the-art word recognition – synthetic data generation followed by whole word input CNNs. We considered three models within this framework, each with a different method for recognising text, and demonstrated the vastly superior performance of these systems on standard datasets. In addition, we introduced a new synthetic word dataset, orders of magnitude larger than any released before.

# Acknowledgements

This work was supported by the EPSRC and ERC grant VisRec no. 228180. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the GPUs used for this research.


abstract: Recent advances in training deep (multi-layer) architectures have inspired a renaissance in neural network use. For example, deep convolutional networks are becoming the default option for difficult tasks on large datasets, such as image and speech recognition. However, here we show that error rates below 1% on the MNIST handwritten digit benchmark can be replicated with shallow non-convolutional neural networks. This is achieved by training such networks using the 'Extreme Learning Machine' (ELM) approach, which also enables a very rapid training time ($\sim$`<!-- -->`{=html}10 minutes). Adding distortions, as is common practise for MNIST, reduces error rates even further. Our methods are also shown to be capable of achieving less than 5.5% error rates on the NORB image database. To achieve these results, we introduce several enhancements to the standard ELM algorithm, which individually and in combination can significantly improve performance. The main innovation is to ensure each hidden-unit operates only on a randomly sized and positioned patch of each image. This form of random 'receptive field' sampling of the input ensures the input weight matrix is sparse, with about 90% of weights equal to zero. Furthermore, combining our methods with a small number of iterations of a single-batch backpropagation method can significantly reduce the number of hidden-units required to achieve a particular performance. Our close to state-of-the-art results for MNIST and NORB suggest that the ease of use and accuracy of the ELM algorithm for designing a single-hidden-layer neural network classifier should cause it to be given greater consideration either as a standalone method for simpler problems, or as the final classification stage in deep neural networks applied to more difficult problems.
author: Mark D. McDonnell; Migel D. Tissera; Tony Vladusich; André van Schaik; Jonathan Tapson
date: 2024-10-01
title: Fast, simple and accurate handwritten digit classification by training shallow neural network classifiers with the 'extreme learning machine' algorithm

# Introduction

The current renaissance in the field of neural networks is a direct result of the success of various types of deep network in tackling difficult classification and regression problems on large datasets. It may be said to have been initiated by the development of Convolutional Neural Networks (CNN) by LeCun and colleagues in the late 1990s  and to have been given enormous impetus by the work of Hinton and colleagues on Deep Belief Networks (DBN) during the last decade . It would be reasonable to say that deep networks are now considered to be a default option for machine learning on large datasets.

The initial excitement over CNN and DBN methods was triggered by their success on the MNIST handwritten digit recognition problem , which was for several years the standard benchmark problem for hard, large dataset machine learning. A high accuracy on MNIST is regarded as a basic requirement for credibility in a classification algorithm. Both CNN and DBN methods were notable, when first published, for posting the best results up to that respective time on the MNIST problem.

The standardised MNIST database consists of 70,000 images, each of size 28 by 28 greyscale pixels . There is a standard set of 60,000 training images and a standard set of 10,000 test images, and numerous papers report results of new algorithms applied to these 10,000 test images, e.g. .

In this report, we introduce variations of the Extreme Learning Machine algorithm  and report their performance on the MNIST test set. These results are equivalent or superior to the original results achieved by CNN and DBN on this problem, and are achieved with significantly lower network and training complexity. This poses the important question as to whether the ELM training algorithm should be a more popular choice for this type of problem, and a more commonplace algorithm as a first step in machine learning.

Table  summarises our results, and shows some comparison points with results obtained by other methods in the past (note that only previous results that do not use data augmentation methods are shown, and only one of our new results is for such a case). Our new results surpass results using earlier deep networks, but recent regularisation methods such as drop connect , stochastic pooling , dropout  and so-called 'deeply supervised networks'  have enabled deep convolutional networks to set new state-of-the-art performance for MNIST for the case where no data-augmentation is used. Nevertheless, our best result for a much simpler single-hidden-layer neural network classifier trained using the very fast ELM algorithm, and without using data augmentation, is within just 41 errors out of 10000 test images of the state-of-the-art.

## The Extreme Learning Machine: Notation and Training

The Extreme Learning Machine (ELM) training algorithm  is relevant for a single hidden layer feedforward network (SLFN), similar to a standard neural network. However, there are three key departures from conventional SLFNs. These are (i) that the hidden layer is frequently very much larger than a neural network trained using backpropagation; (ii) the weights from the input to the hidden layer neurons are randomly initialised and are fixed thereafter (i.e., they are not trained); and (iii) the output neurons are linear rather than sigmoidal in response, allowing the output weights to be solved by least squares regression. These attributes have also been combined in learning systems several times previously .

The standard ELM algorithm can provide very good results in machine learning problems requiring classification or regression (function optimization); in this paper we demonstrate that it provides an accuracy on the MNIST problem superior to prior reported results for similarly-sized SLFN networks .

We begin by introducing three parameters that define the dimensions of an ELM used as an $N$-category classifier: $L$ is the dimension of input vectors, $M$ is the number of hidden layer units, and $N$ is the number of distinct labels for training samples. For the case of classifying $P$ test vectors, it is convenient to define the following matrices:

-   ${\bf X_{\rm test}}$, of size $L\times P$, is formed by setting each column to equal a single test vector.

-   ${\bf Y}_{\rm test}$, of size $N\times P$, numerically represents the prediction vector of the classifier.

To map from input vectors to network outputs, two weights matrices are required:

-   ${\bf W_{\rm in}}$, of size $M\times L$, contains the input weight matrix that maps length-$L$ input vectors to length $M$ hidden-unit inputs.

-   ${\bf W_{\rm out}}$, of size $N\times M$, contains the output weights that project from the $M$ hidden-unit activations to a length $N$ class prediction vector.

We also introduce matrices to describe inputs and outputs to/from the hidden-units:

-   ${\bf D}_{\rm test}:= {\bf W_{\rm in}}{\bf X_{\rm test}}$, of size $M\times P$, contains the linear projections of the input vectors that are inputs to each of the $M$ hidden-units. A bias for each hidden unit can be added by expanding the size of the input dimension from $L$ to $L+1$, and setting the additional input element to always be unity for all training and test data, with the bias values included as an additional column in ${\bf W}_{\rm in}$.

-   ${\bf A}_{\rm test}$, of size $M\times P$, contains the hidden-unit activations that occur due to each training vector, and is given by

$$\begin{aligned}
{\bf A}_{\rm test} := f({\bf D}_{\rm test}),
\end{aligned}$$ where $f(\cdot)$ is shorthand notation for the fact that each element of ${\bf D}_{\rm test}$ is nonlinearly converted term-by-term to the corresponding element of ${\bf A}_{\rm test}$. For example, if the hidden unit response is given by the logistic sigmoid function, then $$\begin{aligned}
({\bf A}_{\rm test})_{i,j} &= f(({\bf D}_{\rm test})_{i,j})=\frac{1}{1+\exp{(-({\bf D}_{\rm test})_{i,j})}}.\label{logistic}
\end{aligned}$$ Many nonlinear activation functions can be equally effective, such as the rectified linear unit (ReLU) function , the absolute value function or the quadratic function. As with standard artificial neural networks, the utility of the nonlinearity is that it introduces hidden-unit responses that represent correlations or 'interactions' between input elements, rather than simple linear combinations of them.

The overall conversion of test data to prediction vectors can be written as $$\begin{aligned}
{\bf Y}_{\rm test}
&= {\bf W}_{\rm out}f({\bf W}_{\rm in}{\bf X}_{\rm test}).
\end{aligned}$$

We now describe the ELM training algorithm. We introduce $K$ to denote the number of training vectors available. It is convenient to introduce the following matrices that are relevant for training an ELM: ${\bf X}_{\rm train}$, of size $L\times K$, ${\bf A}_{\rm train}$, of size $M\times K$, and ${\bf Y}_{\rm train}={\bf W}_{\rm out}{\bf A}_{\rm train}$, of size $N\times K$ are defined analogously to ${\bf X}_{\rm test}$, ${\bf A}_{\rm test}$ and ${\bf Y}_{\rm test}$ above. We also introduce ${\bf Y}_{\rm label}$, of size $N\times K$, which numerically represents the labels of each class of each training vector; it is convenient to define this mathematically such that each column has a $1$ in a single row, and all other entries are zero. The only 1 entry in each column occurs in the row corresponding to the label class for each training vector.

Ideally we seek to find to find ${\bf W_{\rm out}}$ that satisfies $$\begin{aligned}
\label{T}
{\bf Y}_{\rm label}={\bf W}_{\rm out}{\bf A}_{\rm train}.
\end{aligned}$$ However, the number of unknown variables in ${\bf W}_{\rm out}$ is $NM$, and the number of equations is $NK$. Although an exact solution potentially exists if $M=K$, it is usually the case that $M< K$ (i.e., there are many more training samples than hidden units) so that the system is overcomplete. The usual approach then, is to seek the solution that minimises the mean square error between ${\bf {Y}_{\rm label}}$ and ${\bf {Y}_{\rm train}}$. This is a standard least squares regression problem for which the exact solution is ${\bf W}_{\rm out} = {\bf Y}_{\rm label}{\bf A}_{\rm train}^{\top}({\bf A}_{\rm train}{\bf A}_{\rm train}^{\top})^{-1}$, assuming that the inverse exists (in practice it usually does).

It can also be useful to regularise the problem to reduce overfitting, by ensuring that the weights of ${\bf W}_{\rm out}$ do not become large. The standard ridge-regression approach  produces the following closed form solution for the output weights: $$\begin{aligned}
\label{Q}
{\bf W}_{\rm out} &= {\bf Y}_{\rm label}{\bf A}_{\rm train}^{\top}({\bf A}_{\rm train}{\bf A}_{\rm train}^{\top}+c{\bf I})^{-1},
\end{aligned}$$ where ${\bf I}$ is the $M\times M$ identity matrix, and $c$ can be optimised using cross-validation techniques. As is discussed in more detail below, we have found QR decomposition  to be the most effective method for solving for ${\bf W}_{\rm out}$.

# Faster and more accurate performance by shaping the input weights non-randomly

In the conventional ELM algorithm, the input weights are randomly chosen, typically from a continuous uniform distribution on the interval $[-1,1]$ , but we have found that other distributions such as bipolar binary values from $\{-1,1\}$ are equally effective.

Beyond such simple randomisation of the input weights, small improvements can be made by ensuring the rows of ${\bf W}_{\rm in}$ are as mutually orthogonal as possible . This cannot be achieved exactly unless $M \le L$, but simple random weights typically produce dot products of distinct rows of ${\bf W}_{\rm in}$ that are close to zero, albeit not exactly zero, while a dot product of each row with itself is always much larger than zero. In addition, it can be beneficial to normalise the length of each row of ${\bf W}_{\rm in}$, as occurs in the orthogonal case.

In contrast, we can also aim to find weights that, rather than being selected from a random distribution, are instead chosen to be well *matched* to the statistics of the data, with the hope that this will improve generalisation of the classifier. Ideally we do not want to have to learn these weights, but rather just form the weights as a simple function of the data.

Here we focus primarily on improving the performance of the ELM algorithm by biasing the selection of input layer weights in six different ways, several of which were recently introduced in the literature, and several of which are novel in this paper. These methods are as follows:

1.  Select input layer weights that are random, but biased using the training data samples, so that the dot product between weights and training data samples is likely to be large. This is called Computed Input Weights ELM (CIW-ELM) .

2.  Ensure input weights are constrained to a set of difference vectors of between-class samples in the training data. This is called Constrained ELM (C-ELM) .

3.  Restrict the weights for each hidden layer neuron to be non-zero only for a small, random rectangular patch of the input visual field; we call this Receptive Field ELM (RF-ELM). Although we believe this method to be new to ELM approaches, it is inspired by other machine learning approaches that aim to mimic cortical neurons that have limited visual receptive fields, such as convolutional neural networks .

4.  Combine RF-ELM with CIW-ELM, or RF-ELM with C-ELM; we show below that the combination is superior to any of the three methods individually.

5.  Pass the results of a RF-CIW-ELM and a RF-C-ELM into a third standard ELM (thus producing a two-layer ELM system); we show below that this gives the best overall performance of all methods considered in this paper.

6.  Application of the backpropagation method of . This method highlights that the performance of an ELM can be enhanced by adjusting all input layer weights simultaneously, based on all training data. The output layer weights are maintained in their least-squares optimal state by recalculating them after input layer backpropagation updates. The process of backpropagation updating of the input weights followed by standard ELM recalculation of the output weights can be repeated iteratively until convergence.

RF-ELM and its combination with CIW-ELM and C-ELM, and the two-layer ELM are reported here for the first time. We demonstrate below that each of these methods independently improves the performance of the basic ELM algorithm, and in combination they produce results equivalent to many deep networks on the MNIST problem . First, however, we now describe each method in detail.

## Computed Input Weights for ELM

The CIW-ELM approach is motivated by considering the standard backpropagation algorithm . A feature of weight-learning algorithms is that they operate by adding to the weights some proportion of the training samples, or a linear sum or difference of training samples. In other words, apart from a possible random initialization, the weights are constrained to take final values which are drawn from a space defined in terms of linear combinations of the input training data as basis vectors—see Fig. . While it has been argued, not without reason, that it is a strength of ELM that it is not thusly constrained , the use of this basis as a constraint on input weights will bias the ELM network towards a conventional (backpropagation) solution.

The CIW-ELM algorithm is as follows :

1.  For use in the following steps only, normalize all training data by subtracting the mean over all training points and dimensions and then dividing by the standard deviation.

2.  Divide the $M$ hidden layer neurons into $N$ blocks, one for each of $N$ output classes; for data sets where the number of training data samples for each class are equal, the block size is $M_n = M/N$. We denote the number of training samples per class as $K_n,~n=1,\dots,N$. If the training data sets for each class are not of equal size, the block size can be adjusted to be proportional to the data set size.

3.  For each block, generate a random sign ($\pm 1$) matrix, ${\bf R}_n$ of size $M_n \times K_n$.

4.  Multiply ${\bf R}_n$ by the transpose of the input training data set for that class, $X_{{\rm train},n}^{\top}$, to produce $M_n \times L$ summed inner products, which are the weights for that block of hidden units.

5.  Concatenate these $N$ blocks of weights for each class into the $M\times L$ input weight matrix ${\bf W_{\rm in}}$.

6.  Normalize each row of the input weight matrix, ${\bf W_{\rm in}}$, to unity length.

7.  Solve for the output weights of the ELM using standard ELM methods described above.

## Constrained Weights for ELM

Recently, Zhu *et al.*  have published a method for constraining the input weights of ELM to the set of difference vectors of between-class samples. The difference vectors of between-class samples are the set of vectors connecting samples of one class with samples of a different class, in the sample space—see Fig. . In addition, a methodology is proposed for eliminating from this set the vectors of potentially overlapping spaces (effectively, the shorter vectors) and for reducing the use of near-parallel vectors, in order to more uniformly sample the weight space.

The Constrained ELM (C-ELM) algorithm we used is as follows :

1.  Randomly select $M$ distinct pairs of training data such that:

    1.  each pair comes from two distinct classes;

    2.  the vector length of the difference between the pairs is smaller than some constant, $\epsilon$.

2.  Set each row of the $M\times L$ input weight matrix ${\bf W_{\rm in}}$ to be equal to the difference between each pair of randomly selected training data.

3.  Set the bias for each hidden unit equal to the scalar product of the sum of each pair of randomly selected training data and the difference of each pair of randomly selected training data.

4.  Normalize each row of the input weight matrix, ${\bf W_{\rm in}}$, and each bias value, by the vector of the difference of the corresponding pair of randomly selected training data.

5.  Solve for the output weights of the ELM using standard ELM methods described above.

## Receptive Fields for ELM

We have found that a data-blind (unsupervised) manipulation of the input weights improves generalization performance. The approach has the added bonus that the input weight matrix is sparse, with a very high percentage of zero entries, which could be advantageous for hardware implementations, or if sparse matrix storage methods are used in software.

The RF-ELM approach is inspired by neurobiology, and strongly resembles some other machine learning approaches . Biological sensory neurons tend to be tuned with preferred receptive fields so that they receive input only from a subset of the overall input space. The region of responsiveness tends to be contiguous in some pertinent dimension, such as space for the visual and touch systems, and frequency for the auditory system. Interestingly, this contiguity aspect may be lost beyond the earliest neural layers, if features are combined randomly.

In order to loosely mimic this organisation of biological sensory systems, in this paper where we consider only image classification tasks, for each hidden unit we create randomly positioned and sized rectangular masks that are smaller than the overall image. These masks ensure only a small subset of the length-$L$ input data vectors influence any given hidden unit—see Fig. .

The algorithm for generating these 'receptive-field' masks is as follows:

1.  Generate a random input weight matrix ${\bf W}$ (or instead start with a CIW-ELM or C-ELM input weight matrix).

2.  For each of $M$ hidden units, select two pairs of distinct random integers from $\{1,2,\dots L\}$ to form the coordinates of a rectangular mask.

3.  If any mask has total area smaller than some value $q$ then discard and repeat.

4.  Set the entries of a $\sqrt{L}\times \sqrt{L}$ square matrix that are defined by the two pairs of integers to $1$, and all other entries to zero.

5.  Flatten each receptive field matrix into a length $L$ vector where each entry corresponds to the same pixel as the entry in the data vectors ${\bf X}_{\rm test}$ or ${\bf X}_{\rm train}$.

6.  Concatenate the resulting $M$ vectors into a receptive field matrix ${\bf F}$ of size $M\times L$.

7.  Generate the ELM input weight matrix by finding the Hadamard product (term by term multiplication) ${\bf W}_{\rm in}={\bf F} \circ {\bf W}$.

8.  Normalize each row of the input weight matrix, ${\bf W_{\rm in}}$, to unity length.

9.  Solve for the output weights of the ELM using standard ELM methods described above.

We have additionally found it beneficial to exclude pixels from the mask if most or all training images have identical values for those regions. For the MNIST database, this typically means ensuring all masks exclude the first and last 3 rows and first and last 3 columns. For MNIST we have found that a reasonable value of the minimum mask size is $q=10$, which enables masks of size $1\times 10$ and $2 \times 5$ and larger, but not $3 \times 3$ or smaller.

Note that unlike the random receptive field patches described in  for use in a convolutional network, our receptive field masks are not of uniform size, and not square; we found it beneficial to ensure a large range of receptive field areas, and large diversity in ratios of lengths to widths.

## Combining RF-ELM with CIW-ELM and C-ELM

All three approaches described so far provide weightings for pixels for each hidden layer unit. CIW-ELM and C-ELM weight the pixels to bias hidden-units towards a larger response for training data from a specific class. The sparse weightings provided by RF-ELM bias hidden-units to respond to pixels from specific parts of the image.

We have found that enhanced classification performance can be achieved by combining the shaped weights obtained by either CIW-ELM or C-ELM with the receptive field masks provided by RF-ELM. The algorithm for either RF-CIW-ELM or RF-C-ELM is as follows.

1.  Follow the first 5 steps of the above CIW-ELM or the first 2 steps of the C-ELM algorithm, to obtain an un-normalized shaped input weight matrix, ${\bf W_{{\rm in},s}}$.

2.  Follow the first 6 steps of the RF-ELM algorithm to obtain a receptive field matrix, ${\bf F}$.

3.  Generate the ELM input weight matrix by finding the Hadamard product (term by term multiplication) ${\bf W}_{\rm in}={\bf F} \circ {\bf W_{{\rm in},s}}$.

4.  Normalize each row of the input weight matrix, ${\bf W_{\rm in}}$, to unity length.

5.  If RF-C-ELM, produce the biases according to steps 3 and 4 of the C-ELM algorithm, but use the masked difference vectors rather than the unmasked ones.

6.  Solve for the output weights of the ELM using standard ELM methods described above.

## Combining RF-C-ELM with RF-CIW-ELM in a two-layer ELM: RF-CIW-C-ELM

We have found that results obtained with RF-C-ELM and RF-CIW-ELM are similar in terms of error percentage when applied to the MNIST benchmark, but the errors follow different patterns. As such, a combination of the two methods seemed to offer promise. We have combined the two methods using a multiple-layer ELM which consists of an RF-C-ELM network and a RF-CIW-ELM network in parallel, as the first two layers. The outputs of these two networks are then combined using a further ELM network, which can be thought of as an ELM-autoencoder, albeit that it has twenty input neurons and ten output neurons; the input neurons are effectively two sets of the same ten labels. The structure is shown in Fig. . The two input networks are first trained to completion in the usual way, then the autoencoder layer is trained using the outputs of the input networks as its input, and the correct labels as the outputs. The result of this second-layer network, which is very quick to implement (as it uses a hidden layer of typically only 500-1000 neurons), is significantly better than the two input networks (see Table I). Note that the middlemost layer shown in Fig.  consists of linear neurons, and therefore it can be removed by combining its input and output weights into one connecting weight matrix. However, it is computationally disadvantageous to do so because the number of multiplications will increase.

## Fine Tuning by Backpropagation

In all of the variations of ELM described in this report, the hidden layer weights are not updated or trained according to the output error, and the output layer weights are solved using least squares regression. This considerably reduces the trainability of the network, as the number of free parameters is restricted to the output layer weights, which are generally $\sim10^4$ in number. It has been argued that any network in which the total number of weights in the output layer is less than the number of training points will likely be enhanced by using backpropagation to train weights in previous layers . Hence, following the example of deep networks, and specific ELM versions of backpropagation , we have experimented by using backpropagation to fine-tune the hidden layer weights. This does re-introduce the possibility of overfitting, but that is a well-understood problem in neural networks and the usual methods for avoiding it will apply here. For simplicity, a batch mode backpropagation was implemented, using the following algorithm. Note that as in Eqn. (), we assume a logistic activation function in the hidden layer neurons, for which the derivative can be expressed as $f' = f(1-f)$.

1.  Construct the ELM and solve for the output layer weights ${\bf W}_{\rm out}$ as described above.

2.  Perform iterative backpropagation as follows:

    1.  Compute the error for the whole training set: ${\bf E} = {\bf Y}_{\rm label} - {\bf W}_{\rm out}{\bf A}_{\rm train}$.

    2.  Calculate the weights update, as derived by : $\Delta{\bf W}_{\rm in} = \xi\left[({\bf W}_{\rm out}^{\top}{\bf E})\circ({\bf A}_{\rm train}-{\bf A}_{\rm train}\circ{\bf A}_{\rm train})\right]{\bf X}_{\rm train}^{\top}$ where $\xi$ is the learning rate, and $\circ$ indicates the Hadamard product (elementwise matrix multiplication).

    3.  Update the weights, ${\bf W}_{\rm in} = {\bf W}_{\rm in} - \Delta{\bf W}_{\rm in}$.

    4.  Re-calculate ${\bf A}_{\rm train}$ with the new ${\bf W}_{\rm in}$.

    5.  Re-solve for ${\bf W}_{\rm out}$ using least squares regression and continue.

    6.  Repeat from step a) for a desired number of iterations or until convergence.

As illustrated in the Results section, this process has shown a robust improvement on all of the SLFN ELM solutions tested here, provided learning rates which maintained stability were used.

# Results and discussion for the MNIST benchmark

## SLFN with shaped input-weights

We trained ELMs using each of the six input-weight shaping methods described above, as well as a standard ELM with binary bipolar ($\{-1,1\}$) random input weights plus row-normalisation. Following the normalisation of rows of the input weight matrix to unity, we multiplied the values of the entire input weight matrix by a factor of 2, for all seven methods, as this scaling was found to be close to optimal in most cases.

Our results are shown in Fig. . To obtain an indication of variance resulting from the randomness inherent in each input-weight shaping method, we trained 10 ELMs using each method, and then plotted the ensemble mean as a function of hidden-layer size, $M$. We also plotted (see markers) the actual error rates for each trained network. It can be seen in Fig. A that the error rate decreases approximately log-linearly with the log of $M$ for small $M$, before slowing as $M$ approaches about $10^4$. Fig. B shows the error rate when the actual training data is used. Since our best test results occur when the error rate on the training data is smaller than $0.2$%, and we found no significant test error improvement for larger $M$ (see Figs C and D) we conclude that increasing $M$ further than shown here produces over fitting. This can be verified by cross-validation on the training data.

## ELM with shaped input-weights and backpropagation

We also trained networks using each of the methods described in the previous section, plus 10 iterations of ELM-backpropagation using a learning rate of $\xi = 0.001$. As can be inferred from the fact that the training set still has relatively high error rates for most $M$ (Figure B), this use of backpropagation is far from optimal, and does not give converged results. As shown in Figure A, in comparison with Figure A, these 10 iterations at a fixed learning rate still provide a significant improvement in the error rate for small $M$. On the other hand, the improvement for $M=12800$ is minimal, which is not surprising given that the error rate on the training data without using backpropagation for this value of $M$ is already well over 99% (Figure B).

It is likely that we can get further enhancements of our error rates by optimising the backpropagation learning rate and increasing the number of iterations used. Moreover, several methods for accelerating convergence when carrying out backpropagation have been described previously , and we have not used those methods here. However, the best error rate result reported previously for those methods applied to the MNIST benchmark was 1.45%, achieved with 2048 hidden units, and the best error rate for the backpropagation method we used is 3.73% . Our results for $M\ge 3200$ hidden units and RF-C-ELM-BP or RF-CIW-ELM-CP match or surpass the former result, despite using the least advanced backpropagation method in prior work . The latter figure of 3.73% is easily surpassed by all 7 methods for just 800 hidden units. Moreover, the training time reported in prior work  is significantly slower than the time we required when using shaped input weights (see the following section). These outcomes indicate that the use of input-weight shaping prior to backpropagation outperforms backpropagation alone by a large margin, in terms of both error rate and training time.

## Runtime efficiency

In many applications, the time required to train a network is not considered as important as the time required for the network to classify new data. However, there do exist applications in which the statistics of training data change rapidly, meaning retraining is required, or deployment of a trained classifier is required very rapidly after data gathering. For example, in financial, sports, or medical data analysis, deployment of a newly trained classifier can be required within minutes of acquiring data, or retraining may be required periodically, e.g. hourly. Hence, we emphasize in this paper the rapidity of training. The speed for testing is negligible in comparison.

The mean training runtime for each of our methods is shown in Figure . They were obtained using Matlab running on a Macbook pro with a 3 GHz Intel Core i7 (2 dual cores, for a total of 4 cores), running OS X 10.8.5 with 8 GB of RAM. The times plotted in Figure  are the total times for setup and training, excluding time required to load the MNIST data into memory from files. The version of Matlab we used by default exploits all four CPU cores for matrix multiplication and least squares regression. Note that the differences in run time for each method are negligible, which is expected, since the most time-consuming part is the formation of the matrix ${\bf A}_{\rm train}{\bf A}_{\rm train}^{\top}$. The time for testing was not included in the shown data. We found, predictably, that this scaled linearly with $M$, and was about 10 seconds for $M=12,800$.

The most important conclusion we draw in terms of runtime is that our best results shown here (for $M=15000$ hidden units) for RF-CIW-ELM or RF-C-ELM individually take in the order of 15 minutes total runtime and achieve $\sim$`<!-- -->`{=html}99% correct classification on MNIST. In comparison, data tabulated previously for backpropagation shows at least 81 minutes in order to achieve 98% accuracy, and a best result of 98.55% in 98 minutes . In contrast, runtimes reported for the standard ELM algorithm previously (28 seconds for 2048 hidden units ) are comparable to ours (12 seconds for 1600 hidden units and less than 1 minute for 3200 hidden units). This illustrates that improving error rate by shaping the input weights as we have done here has substantial benefits for runtime and error rate in comparison to backpropagation.

## Distorting and pre-processing the training set

Many other approaches to classifying MNIST handwritten digits improve their error rates by preprocessing it and/or by expanding the size of the training set by applying affine (translation, rotation and shearing), and elastic distortions of the standard set . We have also experimented with distorting the training set to improve on the error rates reported here. For example, with 1 and 2 pixel affine translations, we are able to achieve error rates smaller than 0.8%. When we added random rotations, scalings, shears and elastic distortions, we achieved a best repeatable error rate of 0.62%, and an overall best error rate of 0.57%. However, adding distortions of the training set substantially increases the runtime for two reasons. First, more training points generally requires a larger hidden layer size. For example, when we increase the size of the training set by a factor of 10, we have found we need $M>20000$ to achieve error rates smaller than 0.7%. This significantly affects the run time through the $O(M^2)$ matrix multiplication required.

At this stage, we have chosen to not systematically continue to improve the way in which we implement distortions to approach state of the art MNIST results, but our preliminary results show that ELM training is capable of using such methods to enhance error rate performance, at the expense of a significant increase in runtime, as is expected in other non-ELM methods.

## Results on NORB

We briefly present some results on a second well-known image classification benchmark: the NORB-small database . This database consists of 48600 stereo greyscale images from five classes, and there is a standard set of 24300 stereo images for training, and a standard set of 24300 for testing. Each of the images in the two stereo channels for each sample is of size 96$\times$`<!-- -->`{=html}96 pixels.

Given the large size of each image relative to MNIST images, we preprocessed all images by spatially lowpass filtering using a $9\times9$ pixel Gaussian kernel, with standard deviation of 4, and then decimating to $13\times 13$ pixel images. We then contrast-normalised each image by subtracting its mean, and dividing by its standard deviation.

Figure  shows results for the error rate (ten repeats and the ensemble mean are shown) on the test set from application of the RF-C-ELM method, as the number of hidden units increases. We set the minimum receptive field size to 1, and the ridge regression parameter to $c=5\times 10^{-6}$. Our test results peak at close to 95% correct, which is within 3% of state-of-the-art , and superior to some results for deep convolutional networks .

## Computationally efficient methods for ELM training: iterative methods for large training sets

In practice, it is known to be generally computationally more efficient (and avoids other potential problems, such as those described in ) to avoid explicit calculation of matrix inverses or pseudo-inverses when solving linear sets of equations. The same principle applies when using the ELM training algorithm, and hence, it is preferable to avoid explicit calculation of the inverse in Eqn. (), and instead treat the following as a set of $NM$ linear equations to be solved for $NM$ unknown variables: $$\begin{aligned}
\label{Final}
{\bf Y}_{\rm label}{\bf A}_{\rm train}^{\top} &= {\bf W}_{\rm out}({\bf A}_{\rm train}{\bf A}_{\rm train}^{\top}+c{\bf I}).
\end{aligned}$$ Fast methods for solving such equations exist, such as the QR decomposition method , which we used here. For large $M$, the memory and computational speed bottleneck in an implementation then becomes the large matrix multiplication, ${\bf A}_{\rm train}{\bf A}_{\rm train}^{\top}$. However, there are simple methods that can still enable solution of Eqn. () when $M$ is too large for this multiplication to be carried out in one calculation.

For example, when solving Equation () by implementation in MATLAB, it is computationally efficient to use the overloaded '$\backslash$' function, which invokes the QR decomposition method. This approach can be used either for the inverse or pseudo-inverse, but we have found it faster to solve (), which requires the inverse rather than the pseudo-inverse.

Well known software packages such as MATLAB (which we used) automatically exploit multiple CPU cores available in most modern PCs to speed up execution of this algorithm using multithreading. Alternative methods like explicitly calculating the pseudo-inverse, or singular value decomposition, are in comparison significantly (sometimes several orders of magnitude) slower.

When using the linear equation solution method, the main component of training runtime for large hidden-layer sizes becomes the large matrix multiplication required to obtain ${\bf A}_{\rm train}{\bf A}_{\rm train}^{\top}$. There is clearly much potential for speeding up this simple but time-consuming operation, such as by using GPUs or other hardware acceleration methods.

The above text discusses the standard single-batch approach. There are also online and incremental ELM solutions for real-time and streaming operations and large data sets . The use of singular value decomposition offers some additional insight into network structure and further optimization . Here we describe an iterative method that offers advantages in training where the output weight matrix need not be calculated more than once.

One potential drawback of following the standard ELM method of solving for the output weights using all training data in one batch is the large amount of memory that is potentially required. For example, with the MNIST training set of 60000 images, and a hidden layer size of $M=10000$, the ${\bf A}_{\rm train}$ matrix has $6\times10^8$ elements, which for double precision representations requires approximately $4.5$ GB of RAM. Although this is typically available in modern PCs, the amount of memory required becomes problematic if training data is enhanced by distortions, or if the amount of hidden units needs to be increased significantly.

We have identified a simple solution to this problem, which is as follows. First, we introduce size $M\times 1$ vectors ${\bf d}_j,~j=1,\dots,K$, formed from the columns of ${\bf A}_{\rm train}$. Then one of the two key terms in Eqn. () can be expressed as $$\begin{aligned}
\label{FinalA}
{\bf A}_{\rm train}{\bf A}_{\rm train}^{\top}= \sum_{j=1}^K {\bf d}_j{\bf d}_j^{\top}.
\end{aligned}$$ That is, the matrix that describes correlations between the activations of each hidden unit is just the sum of the outer products of the hidden-unit activations in response to all training data. Similarly, we can simplify the other key term in Eqn. () by introducing size $N\times 1$ vectors ${\bf y}_j,~j=1,\dots K$ to represent the $K$ columns of ${\bf Y}_{\rm label}$ and write $$\begin{aligned}
\label{FinalA2}
{\bf Y}_{\rm label}{\bf A}_{\rm train}^{\top}= \sum_{j=1}^K {\bf y}_j{\bf d}_j^{\top}.
\end{aligned}$$ In this way, the $M\times M$ matrix ${\bf A}_{\rm train}{\bf A}_{\rm train}^{\top}$ and the $N\times M$ matrix ${\bf Y}_{\rm label}{\bf A}_{\rm train}^{\top}$ can be formed from $K$ training points without need to keep the ${\bf A}_{\rm train}$ matrix in memory, and once these are formed, the least squares solution method applied. The matrix ${\bf A}_{\rm train}{\bf A}_{\rm train}^{\top}$ still requires a large amount of memory ($M=12000$ requires over 1 GB of RAM), but using this method the number of training points can be greatly expanded and incur only a runtime cost. In practice, rather than form the sum from $K$ training points, it is more efficient to form batches of subsets of training points and then form the sums: the size of the batch is determined by the maximum RAM available.

It is important to emphasise that unlike other iterative methods for training ELMs that update the output weights iteratively , the approach described here only iteratively updates ${\bf A}_{\rm train}{\bf A}_{\rm train}^{\top}$.

# Conclusions

We have shown here that simple SLFNs are capable of achieving the same accuracy as deep belief networks  and convolutional neural networks  on one of the canonical benchmark problems in deep learning: image classification. The most accurate networks we consider here use a combination of several non-iterative learning methods to define a projection from the input space to a hidden layer. The hidden layer output is then solved simply using least squares regression applied to a single batch of all training data to find the weights for a linear output layer. If extremely high accuracy is required, the outputs of one or more of these SLFNs can be combined using a simple autoencoder stage. The maximum accuracy obtained here is comparable with the best published results for the standard MNIST problem, without augmentation of the dataset by preprocessing, warping, noising/denoising or other non-standard modification. The accuracies achieved for the basic SLFN networks are in some cases equal to or higher than those achieved by the best efforts with deep belief networks, for example.

Moreover, when using the receptive field (RF) method to shape inputs weights, the resulting input weight matrix becomes highly sparse: using the RF algorithm above, close to 90% of input weights are exactly zero.

We note also that the implementations here were for the most part carried out on standard desktop PCs and required very little computation in comparison with deep networks. It should be highlighted that we have found significant speed increases for training by avoiding explicit calculation of matrix inverses. Moreover, we have shown that it is possible to circumvent memory difficulties that could arise with large training sets, by iteratively calculating the matrix ${\bf A}_{\rm train}{\bf A}_{\rm train}^\top$, and then still only computing the output weights once. This method could also be used in streaming applications: the matrix ${\bf A}_{\rm train}{\bf A}_{\rm train}^\top$ could be updated with every training sample, but the output weights only updated periodically. In these ways, we can avoid previously identified potential limitations of the ELM training algorithm, regarding matrix inversion discussed in  (see also ).

The principles implemented in the ELM training algorithm, and in particular the use of single-batch least squares regression in a linear output layer, following random projection to nonlinear hidden-units, parallel a principled approach to modelling neurobiological function, known as the *neural engineering framework* (NEF) . Recently this framework  was utilized in a very large (2.5 million neuron) model of the functioning brain, known as SPAUN . The computational and performance advantages we have demonstrated here could potentially boost the performance of the NEF, as well as, of course, the many other applications of neural networks.

Although deep networks and convolutional networks are now standard for hard problems in image and speech processing, their merits were originally argued almost entirely on the basis of their success in classification problems such as MNIST. The argument was of the form that because no other networks were able to achieve the same accuracy, the unique hierarchy of representation of features offered by deep networks, or the convolutional processing offered by CNNs, must therefore be necessary to achieve these accuracies. However, If there exists a neural network that does not use a hierarchical representation of features, and which can obtain the same accuracy as one that does, then this argument may be a case of confirmation bias. We have shown here that results equivalent to those originally obtained with deep networks and CNNs on MNIST can be obtained with simple single-layer feedforward networks, in which there is only one layer of nonlinear processing; and that these results can be obtained with very quick implementations. While the intuitive elegance of deep networks is hard to deny, and the economy of structure of multilayer networks over single layer networks is proven, we would argue that the speed of training and ease of use of ELM-type single layer networks makes them a pragmatic first choice for many real-world machine learning applications.

# Acknowledgments

Mark D. McDonnell's contribution was by supported by an Australian Research Fellowship from the Australian Research Council (project number DP1093425). André van Schaik's contribution was supported by Australian Research Council Discovery Project DP140103001.

# References

# Figures

# Tables

| Grouping | Method | Error in testing | Reference |
|:--:|:--:|:--:|:--:|
| Selected Non-ELM | SLFN, 784-1000-10 | 4.5% |  |
|  | Deep Belief Network | 1.25% |  |
|  | Deep Conv. Net LeNet-5 | 0.95% |  |
|  | Deep Conv. Net (dropconnect) | 0.57% |  |
|  | Deep Conv. Net (stochastic pooling) | 0.47% |  |
|  | Deep Conv. Net (maxout units and dropout) | 0.45% |  |
|  | Deep Conv. Net (deeply-supervised) | 0.39% |  |
| Past ELM | ELM, 784-1000-10 | 6.05% |  |
|  | C-ELM | $\sim$`<!-- -->`{=html}5% |  |
|  | CIW-ELM, 784-1000-10 | 3.55% |  |
|  | ELM, 784-7840-10 | 2.75% |  |
|  | ELM, 784-unknown-10 | 2.61% |  |
|  | CIW-ELM, 784-7000-10 | 1.52% |  |
|  | ELM+backpropagation, 784-2048-10 | 1.45% |  |
|  | Deep ELM, 784-700-15000-10 | 0.97% |  |
| ELM & backprop | RF-(CIW & C)-ELM, 784-(2$\times 6400$)-20-500-10 | 0.91% (1.04%) | This report |
| SLFN ELM | RF-ELM,784-15000-10 | 1.36% (1.48%) | This report |
|  | CIW-ELM,784-15000-10 | 1.28% (1.45%) | This report |
|  | C-ELM, 784-15000-10 | 1.26% (1.33%) | This report |
|  | RF-CIW-ELM , 784-15000-10 | 1.03%(1.10%) | This report |
|  | RF-C-ELM, 784-15000-10 | 0.9% (0.99%) | This report |
|  | RF-C-ELM, 784-25000-10, Distortions | 0.57% (0.62%) | This report |
| DLFN ELM | RF-(CIW& C)-ELM, 784-(2$\times15000$)-20-500-10 | 0.83% (0.87%) | This report |

**Comparison of our results on the MNIST data set with published results using other methods. The percentages listed in brackets are the mean error percentage we obtained from 10 independent realisations of each method. The remaining percentage for the results obtained in this report are from the trained ELM out of the 10 repeats with the best results. Values for each trial shown in Figs  and  demonstrate small spreads either side of the mean value. The abbreviations CIW, C and RF are explained in later sections. Note: SLFN is "Single-hidden Layer Feedforward Network"; DLFN is "Dual-hidden Layer Feedforward Network." The result described as 'Distortions' was obtained by augmenting the training set using affine and elastic distortions, as describe in the main text. 'Deep. Conv. Net' is an abbreviation for 'Deep Convolutional Network.'**