Local Contrast Learning

Chuanyun Xu * 1 Yang Zhang 2 Xin Feng 1 YongXing Ge 3 Yihao Zhang 1 Jianwu Long 1

arXiv:1802.03499v1 [cs.LG] 10 Feb 2018

Abstract
Learning a deep model from small data is yet an
opening and challenging problem. We focus on
one-shot classification by deep learning approach
based on a small quantity of training samples. We
proposed a novel deep learning approach named
Local Contrast Learning (LCL) based on the key
insight about a human cognitive behavior that human recognizes the objects in a specific context by
contrasting the objects in the context or in her/his
memory. LCL is used to train a deep model that
can contrast the recognizing sample with a couple
of contrastive samples randomly drawn and shuffled. On one-shot classification task on Omniglot,
the deep model based LCL with 122 layers and
1.94 millions of parameters, which was trained on
a tiny dataset with only 60 classes and 20 samples
per class, achieved the accuracy 97.99% that outperforms human and state-of-the-art established
by Bayesian Program Learning (BPL) trained on
964 classes. LCL is a fundamental idea which
can be applied to alleviate parametric model’s
overfitting resulted by lack of training samples.

1. Introduction
Deep learning has achieved large successes in many domains of science, business and government for many years
because it can provide a kind of end-to-end approach for
machine learning and requires very little engineering by
hand (LeCun et al., 2015). However, the deep model requires large amount of annotated data for tuning its millions
of parameters. Building a large training set for deep learning is sometimes extremely expensive and not acceptable,
which hinder deep learning to be applied in more domains.
Human can learn a novel concept from just one or few examples (Fei-Fei et al., 2006; Lake et al., 2015a). Can deep
*
Equal contribution 1 College of computer science and engineering,Chongqing University of Technology, Chongqing, China
2
College of Computer and Information Science,Chongqing Normal University, Chongqing, China 3 School of Software Engineering,Chongqing University, Chongqing, China. Correspondence to:
Chuanyun Xu <33677670@QQ.COM>.

model learn a new concept from a small training data, and it
don’t fall into overfitting? This is an opening and challenging question of machine learning. Most of traditional deep
learning approaches try to learn functions that map each
high-dimensional sample to lower-dimensional space with
the least average training error. In nature, they learn some
common and global representations from training data for
detecting or classifying patterns in the input(LeCun et al.,
2015). It is difficult to draw the representation features from
a small training set because the distribution of training data
might not be identical with the test set. When the training
data are sparse and diverse, it becomes more difficult. This
is inconsistent with the human beings cognitive behaviors
that human can more easily distinguish two completely different kinds of objects, such as a car and an apple. So,
we can propose a hypothesis that in the lower cognitive
level, human recognizes the objects in a specific context by
contrasting the objects in the context or in her/his memory
instead of drawing universal representations for building a
global mapping function.
Based on the key insights, the paper introduced the Local
Contrast Learning (LCL) approach for deep learning. LCL
makes use of two key ideas from the human cognitive behavior: (a) Local context:Cognition always base on a local
context, and only depends on the local context. Local context consists of a set of objects drawn from the objects in
global context for cognition. If the context cannot provide
enough information for cognition, the new local context
must be built by gathering new information or recalling new
memories. To human being, the local context must be a
small scenario only with a few target objects. The number
of the targets are usually less than seven(Miller, 1967). (b)
Contrast:In order to identify an object, the recognition is
iteratively executed by contrasting the object and each contrastive object in the local context, and the contrast results of
the different contrastive objects are compared again among
them. The process of the local contrast cognition is shown
in Figure 1.

2. Local Contrast Learning
LCL is a novel machine learning approach that trains a parametric model by contrasting objects in a local context. The
model can learn to distinguish the positive and negative ex-

Local Contrast Learning

Figure 1. The process of the local contrast cognition. (a) The local
contrast context with one recognizing object and five contrastive
objects. (b) The contrastive pairs built by moving attention focused
on every contrastive object. (c) The difference representations
created by contrasting every contrastive pair. (d) Recognition
results got by contrasting the difference representations.

amples in a group of the contrastive samples by contrasting
one by one. LCL firstly generates a large amount of contrastive sample groups by randomly sampling classes and
exemplars from training set and shuffling them, and then
feeds the sample groups into a parametric model to encode
the differences, last outputs the activations of the contrastive
samples by contrasting the differences. LCL can learn recognization in different local contexts rather than a single
global context. The workflow of LCL is shown in Figure 2.
LCL consists of three major components: Contrast Cognitive Context Constructor (CCCC), Difference Embedding
Generator (DEG), and Difference Perceptron (DP).

Figure 2. Workflow of Local Contrastive Learning. (a) Contrasting Cognitive Context Constructor including Category Sampler,
Object Sampler, order Shuffler, (b) Local Cognitive Context, (c)
Contrastive pair, (d) recognizing object, (e) Contrastive Positive
Object, (f) Contrastive Negative Object, (g) Difference Embedding
Generator that is a parametric model with weights shared, (h) Difference Embedding, (i) Difference Perceptron, (j) Contrastive Pair
Local Activation.

to 1). There is only one sample in each category in the contrastive objects.In order to build an LCC, CCCC randomly
samples some classes as the recognizing class and contrast
classes, then randomly samples one or few instances from
each class, and finally randomly shuffles all instances.

CCCC is responsible for constructing the Local Cognitive Context (LCC) from the labelled support set (training set), denoted as S = {S1 , · · · , Sc , · · · , SSC } and
Sc = {(xci , yc )}K
i=1 , where K is the number of the
training sample in each category. Here xci is the ith
training sample in the cth category. Here yc ∈ CS =
{c1 , · · · , cc , · · · , cSC } is the corresponding label and SC
is the numbers of the categories. A LCC consists of one
or few recognizing objects from the same category and a
set of contrastive
objects from differentcategories, denoted

L
as LCC = (xc , yc ), {(xi , yi , zi )}i=1 is defined by a rec-

In LCC, the order of contrastive objects is critical. In other
word, different LCC can be created from the same contrastive objects by organizing different order. A large quantity of LCC can be created from a small quantity of training
samples. For example, in the Small Omniglot dataset with
136 classes, 20 instances each class, in case L equals 20, the
1.18e+44 different LCC can be generated. So, CCCC can
provide a large enough number of LCC for training LCL
model. It is a critical factor to assure that the training of
LCL can stand up to overfitting in facing tiny training set.

ognizing object (xc , yc ), and a set of contrastive objects
L
{(xi , yi , zi )}i=1 . Here L is the numbers of contrastive objects in an LCC. In the contrastive objects, there is one and
only one sample whose category is equal to yc and that is
called contrastive positive object (Corresponding zi is set to
0), but there is no such a sample xi that is equal to xc . The
contrastive objects except this contrastive positive object are
called as contrastive negative object (Corresponding zi is set

DEG is responsible for contrasting the recognizing object with contrastive objects in LCC one by one and
generating a vector representing the differences between them. DEG firstly creates the contrastive pairs
L
{((xc , yc ), (xi , yi , zi ))}i=1 by grouping the recognizing object and each contrastive object in LCC together, and then
the contrastive pairs are iteratively fed into an identical parametric model (such as Resnet(He et al., 2016a;b)) in order

Local Contrast Learning

Algorithm 1 Generate Local Cognitive Context
Input: a labelled training set S, and the number of contrastive objects L
1. Randomly sample L categories from CS as contrastive
categories.
2. Select anyone as contrastive positive category from the
contrastive categories, and others as contrastive negative
categories.
3. Iteratively sample one image from each contrastive
negative categories as contrastive negative objects.
4. Sample two images from the contrastive positive category, among which one is as recognizing object (xc , yc )
and another is as contrastive positive object.
5 Concatenate the contrastive positive objects and
the contrastive negative objects as contrastive objects
L
{(xi , yi , zi )}i=1 .
L
6. Randomly shuffle {(xi , yi , zi )}i=1 .

to get difference embedding that represents the difference
of the contrastive pair.
DE = (DE 1 , · · · , DE i , · · · , DE L )
DE i = deg(((xc , yc ), (xi , yi , zi )))
Here deg is Difference Embedding Generator.
Theoretically, the embedding model may be any kind of
parametric model that can be used as encoder. In the paper,
we use Resnet. It must be emphasized that it is a pair of
objects inputted into embedding model, among which one
is recognizing object and another is contrastive object (positive or negative), and this is like DPSL(Grm et al., 2016),
while this is unlike Siamese Network(Koch et al., 2015;
Kumar et al., 2016), Matching Nets(Vinyals et al., 2016),
or Triplet Network(Kumar et al., 2016; Hoffer & Ailon,
2015), Prototypical Networks(Snell et al., 2017),into which
the objects are inputted one by one. This is important for
avoiding overfitting. In case of 1000 training samples, if
just one object was inputted into embedding model, there
were 1000 different inputs, whereas, if two objects were inputted into embedding model, there were 999,000 different
inputs. From another view to interpret this, the embedding model maps contrastive object from high-dimensional
space to low-dimensional space on condition to recognizing
object. It is more difficult to learn the conditional embedding than one object embedding, so this enforce embedding
model to sense the difference between recognizing object
and contrastive object instead of remembering the object or
its features. Moreover, the embedding on condition to recognizing object is local rather than global. The LCL model
must learn to embed in different LCC rather than whole
training dataset context, so, the model must learn to adapt
the tremendous quantity of context instead of single global

context. It must be noted that DEG outputs an embedding
vector in low dimensional space representing the difference
between recognizing object and the contrastive object, instead of a similarity metric or an embedding representation
of single object for feeding into similarity function(Koch
et al., 2015; Kumar et al., 2016; Vinyals et al., 2016; Hoffer
& Ailon, 2015; Wang et al., 2014). It is more robust to map
high-dimensional object to vector with 64 dimension than
to 1 dimension (In the paper, difference embedding is 64
dimension).
DP is responsible for mapping a group of difference embedding to a vector whose each element represents Contrastive
Pair Local Activation (CPLA).
A = (ai )L
i=1 = dp(DE), ai ∈ R
Here dp is Difference Perceptron and ai is the ith CPLA.
dp is a parametric model, such as fully-connected neural
network. It is emphasized that all DE are inputted into dp
one time, instead of one bye one. ai is not determinated
only by DE i , but by all Difference Embedding.
CPLA indicates the contrastive object in an LCC is positive or negative. In a LCC, the activation of positive object
should be obviously distinguished from the negative object,
however, the value of the activation cannot be directly corresponding to the similarity between the recognizing object
and the contrastive object. The direct rank for the activations in the entire training set is meaningless, because the
contrast results are local in a contrast context. LCL does not
aim to train a global similarity metric for contrastive objects.
Instead, it just locally identifies the positive object in LCC.
It is emphasized that ai is computed from all difference embedding DE of LCC instead of the difference embedding
DE i . That means that CPLA is result of contrasting all Difference Embeddings rather than mapping each Difference
Embedding.

3. Local Contrastive Neural Network
We have provided an implementation of LCL called Local
Contrastive Neural Network (LCNN) for the image classification (see Figure 3). LCNN is a multiple layer deep neural
network using a Resnet as DEG and a fully-connected layer
as DP. The recognizing image and contrastive images in
each contrastive pair of LCC are stacked as the input with
two channels of Resnet, whose all outputs are concatenated
as the input of the fully-connected layer, and its output,
CPLA, is a vector whose element is corresponding to each
contrastive object. The correspondence between CPLA and
contrastive pair is a soft relation which is established by
training. In the paper, we make the activation of the negative object tend to one and the activation of the positive
object tend to zero. Therefore, the contrastive object with
the minimum activation is positive and other are negative.

Local Contrast Learning

In order to tune the parameters of LCNN, we constructed
contrastive loss function (Bell & Bala, 2015; Hadsell et al.,
2006). Minimizing the loss function can enlarge the activation gap between the positive object and the negative
objects.

Figure 3. The network architecture of Local Contrastive Neural
Network. (a) Contrasting Cognitive Context Constructor. (b)
Input included 20 contrastive objects and one recognizing object
from Local Contrastive Context where 28 is image size. (c) 20
Contrastive Pairs. (d) Difference Embedding Generator which is a
Resnet to be iteratively fed a Contrastive Pair. (e) Concatenating
Difference Embedding. (f) Contrastive Pair Local Activation. (g)
Minimum Contrastive Pair Local Activation as the positive object.

Lc (W ) =

N
1 X1
N 1 L

L
X

One-shot recognition is an extreme case. However, in industrial applications, in order to improve classification accuracy,
few-shot recognition is more practicable. The architecture
of a few-shot LCNN is shown in Figure 4. The few-shot
LCC with a few recognizing samples are built, then they are
fed into an identical LCNN for computing CPLA. Finally
all CPLA are summed as few-shot CPLA. Similar one-shot
recognition, the minimum few-shot CPLA is corresponding
the positive object.

Figure 4. The network architecture of LCNN for few-shot. The
figure shows the architecture for 20-way 5-shot classification. (a)
Contrasting Cognitive Context Constructor which build LCC with
few recognizing objects. (b) The few-shot LCC with 5 recognizing
samples. (c) 5 LCC built by unstacking the few-shot LCC. (d) Each
LCC is iteratively inputted into a LCNN for generating CPLA. (e)
All CPLA are summed as CPLA of the few-shot LCC. (f) The
contrastive sample with minimum CPLA is recognized as positive
sample.
!

− (zi log ai + (1 − zi ) log(1 − ai ))

1

Where N is number of LCC in a training mini-batch. L is
number of the contrastive objects in an LCC. ai is CPLA
and zi is expected CPLA. zi equals 0 if the ith contrastive
objects in LCC is a positive object, otherwise zi equals 1.
W are parameters of LCNN including the parameters Wdeg
of the DEG and the parameters Wdp of the DP.
In order to minimize the loss L(W ), the stochastic gradient descent with momentum optimization(Rumelhart et al.;
Sutskever et al.) algorithm is used like Resnet. The momentum of the optimization algorithm is fixed to 0.9. The
learning rate starts from 0.1 and after d1 and d2 iterations,
the learning rate is respectively set to 0.01, 0.001. d1 and
d2 are two hyper parameters. The variance scaling initializer(He et al.) is used for initializing all weight parameters.

4. Experiments
In order to evaluate the one-shot (includes few-shot) classification performance of LCNN, we carried out the classification experiments on two benchmark datasets: Omniglot(Lake et al., 2015a), CASIA-HWDB1.1(Liu et al.,
2011; 2013). According to conventional one-shot test protocol(Lake et al., 2015a), all categories of the datasets are
divided to two subsets: training set (background set) and test
set, and the categories of the training set are disjoint with
the categories of the test set. The model will be trained on
the categories of the training set and tested on the categories
of the test set.
LCNN uses pre-activation Resnet as DEG which are constructed by many pre-activation residual units(He et al.,
2016b). Pre-activation residual unit consists of two activa-

Local Contrast Learning

tion functions (ReLU and BN(Ioffe & Szegedy, 2015)) and
two 3x3 convolutional layers. The first layer of DEG is 3x3
convolutional layer with 16 output channels, followed by n
pre-activation residual units with 16 output channels (Here
n is a hyper-parameter), n units with 32 output channels,n
units with 64 output channels. The last two layers of DEG
are activation function (ReLU and BN) and global average
pooling (Lin et al., 2013) that outputs 64 dimension vector. The number of layers of DEG is determined by hyper
parameter n and the number of the layers of LCNN equals
(n × 2) × 3 + 2.
The learning rate is decayed to 0.01 from 0.1 when the
learning steps is larger than d1 , and then decayed to 0.001
when the learning steps is larger than d2 (Here d1 and d2 are
two hyper-parameters). Maximum training steps m is hyperparameter. If not specified, in the following experiments,
the steps of the learning rate decay d1 and d2 are separately
set to 44800 and 51200, and m is 57600. n is set to 20, that
is to say, the number of the layers of LCNN is 122. The
training mini-batch size is 40.
Every test result performed on same test set might be different due to three kinds of randomness and computational
error, so we run test 100 times on the test set and we report the mean classification accuracy with 95% confidential
interval.
4.1. Omniglot Based BPL Test Protocol
Omniglot dataset consists of images across 1623 classes
with only 20 images per class, from 50 different alphabets
total. The dataset is divided into a background (training
or support) set of 30 alphabets with 964 characters and an
evaluation (test) set of 20 alphabets with 659 characters.
According to the suggestion from Lake et al (Lake et al.,
2015b), only the background set should be used for training
and one-shot learning results are reported using alphabets
from the evaluation set. Some one-shot classification results reported ,such as MANN(Santoro et al., 2016),Matching Nets(Vinyals et al., 2016), Neural Statistician(Edwards
& Storkey, 2017), ConvNet with Memory Module(Kaiser
et al., 2017), Prototypical Networks(Snell et al., 2017),
MAML(Finn et al., 2017), MetaNet(Munkhdalai & Yu,
2017), have not accepted the suggestion, so the results dont
match our results in this section and we do comparisons in
next section with them.
Except the training set of 30 alphabets with 964 characters,
Lake et al(Lake et al., 2015b) have provided smaller background sets ”background small 1” and ”background small
2”. Each of these contains just 5 alphabets and respectively
contains 136 characters and 156 characters. Furthermore,
to see how the models perform with more limited training
classes, we built two tiny subsets by picking up the first
one and first two characters from each alphabet of Omniglot

background set, which are respectively called background
tiny1 and background tiny 2. The two subsets respectively
have 30 characters and 60 characters, and 600 samples and
1200 samples. We trained the LCNN models on all 5 training set for evaluating its performance.
To exactly compare one-shot learning performance, Lake et
al(Lake et al., 2015b; 2013) developed a standard one-shot
test protocol, and we call it BPL test protocol. BPL test
protocol provided 20 groups of within-alphabet one-shot
classification tasks and each task includes 20 characters
selected from an alphabet in the evaluation alphabets. Each
character has two examples drawn by two persons, one as
test example and another as training example. The test task
is, given a test example, to select an example from the 20
training examples, which is from identical character with the
test example. So 400 one-shot test trials can be constructed
from the 20 test tasks and from which the classification
accuracy is calculated. In this section, all experiments are
carried out on the 400 trials.
In the experiments, firstly, using CCCC, a mini-batch of
LCC are constructed from Omniglot training set, which size
is 40 in the all experiments. The number of contrastive
objects, L, in an LCC is 20, so the performance reported is
of 20 way classification. Then the LCC are fed into LCNN
for training. The test LCC are created from previous 400
test trials, one trial to one LCC. We augmented the training
set through 90, 180 and 270 degrees rotations and resized
images to 28 x 28 pixels.
For comparing the performance on small training set with
other neural networks approaches, we selected Matching
Networks(Vinyals et al., 2016) as baseline. We implemented
our own basic version without the fully-conditional embedding (FCE). In the basic version, a convolutional network is
trained to learn independent embeddings for examples in the
training and test set, which consists of a stack of modules,
each of which is a 3x3 convolution with 64 filters followed
by batch normalization(Ioffe & Szegedy, 2015), a Relu nonlinearity and 2x2 max-pooling. Maximum training steps is
120,000. The model were trained on 4 dataset: Omniglot
Tiny1, Omniglot Tiny2, Omniglot Small1, and Omniglot
Small2. The results reported are tested on the previous 400
test trials.
In Table 1, we listed the experiment results along with
the prior models, human performance, and Matching Networks10 based on BPL test protocol.
Using 964 characters for training, LCNN achieved 98.92%
one-shot classification accuracy, and it outperforms the
state-of-the-art accuracy 96.7% established by BPL and
also it is better than humans identification accuracy 95.5%.
The Siamese convolutional network achieved 92.0% accuracy(Koch et al., 2015) that is much worse than LCNN.

Local Contrast Learning
Table 1. One-shot Classification Accuracy (%) on Omniglot Based
BPL Test Protocol. Results are accuracies averaged over 100 runs
and with 95% confidence intervals where reported. The train set
tiny1 and tiny2 consists of first character and first two characters
of Omniglot background set.

M ODEL
ACC .
H UMAN (L AKE ET AL ., 2015 B )
P IXEL K NN(L AKE ET AL ., 2013)
A FFINE MODEL (L AKE ET AL ., 2013)
D EEP B OLTZMANN M ACHINES
(L AKE ET AL ., 2013)
C ONVOLUTIONAL S IAMESE N ET
(KOCH ET AL ., 2015)
M ATCHING N ETS
(V INYALS ET AL ., 2016) TINY 1
M ATCHING N ETS
(V INYALS ET AL ., 2016) TINY 2
M ATCHING N ETS
(V INYALS ET AL ., 2016) SMALL 1
M ATCHING N ETS
(V INYALS ET AL ., 2016) SMALL 2
BPL SMALL 1(L AKE ET AL ., 2013)
BPL SMALL 2(L AKE ET AL ., 2013)
BPL(L AKE ET AL ., 2013)
LCNN TINY 1
LCNN TINY 2
LCNN SMALL 1
LCNN SMALL 2
LCNN

#C HARS

ACC .

964
964

95.5
21.7
81.8

964

62

964

92

30

19.9

tical alphabets, but have different characters. However, in
BPL Test Protocol, the training set and test set have completely different alphabets, that is to say, a model trained
on some alphabets is used to classify on some other alphabets. It is more challenging to recognize the images
in the novel alphabets than in novel characters in identical
alphabets. Some researches also follow Vinyals, such as
such as MANN(Santoro et al., 2016),Matching Nets(Vinyals
et al., 2016), Neural Statistician(Edwards & Storkey, 2017),
ConvNet with Memory Module(Kaiser et al., 2017), Prototypical Networks(Snell et al., 2017), MAML(Finn et al.,
2017), MetaNet(Munkhdalai & Yu, 2017). All test trials are
constructed only from the test set and each trial consists of
a few characters picked from the test set. Only one image
of each character is picked as contrastive object, and one or
a few images are picked as recognizing objects.
MetaNet(Munkhdalai & Yu, 2017) has trained and tested

60

37.56 on BPLs split of 30 training alphabets with 964 classes,

136

47.51 to test the model, so it cannot completely match the BPLs

156
136
156
964
30
60
136
156
964

however it formed 400 trials from the evaluation classes

47.26
95.7
96
96.7
92.82±0.08
97.99±0.05
98.65±0.06
99.17±0.04
98.92±0.04

Using Omniglot background small 1 and Omniglot background small 2 for training, LCNN respectively achieved
accuracies 98.65%, 99.17% that outperformed BPL that
achieved 95.7%,96.0% respectively and also outperformed
Matching Nets(Vinyals et al., 2016) that achieved 47.51%,
47.26%. Furthermore, using Omniglot background tiny1
and Omniglot background tiny2, LCNN achieved 92.82%,
97.99% one-shot classification accuracies. In this case, even
though Matching Nets has just 4 convolutional layers, it
got into serious overfitting, which training accuracies are
100.00% and 99.00%, however test accuracies are 19.90%
and 37.56%. LCNN outperformed BPL just using 6 percent
samples of BPL.

result which be evaluated on standard 400 trials.
In order to extensively evaluate the performances of LCNN
and try to best match other approaches, in this section, we
trained and tested the LCCN separately on the test protocol
provided by Vinyals, O. et al(Vinyals et al., 2016) with 1200
characters and on the test protocol provided by Lake(Lake
et al., 2015b) with 964 characters. Furthermore, in order
to show the advantages of LCNN on small data, we trained
LCNN on the first 60 characters and the first 156 characters
from the 1200 training characters. In order to keep our results reproducible and be available for comparison, unlike
other approaches to randomly split on every run, we used
same the split on every run (The split will be published).
Similar to previous section, we also trained LCNN on ”background small 2” and ”background tiny 2”, but formed trials
from the evaluation set rather than the standard 400 trials
provided by Lake.

4.2. Omniglot Based Variant Test Protocol

In all experiments in this section, the test trials are created
from the test set using Algorithm 1, and the number of test
trials ntrial is computed by the equation ntrial = (EC ×
KE)/(L+nshot ). Here EC is the number of the characters
in the test set. KE is the number of sample per character.
L is the number of contrastive objects in a LCC, and nshot
is the numbers of the shots.

There are a few variant tests proposed by Vinyals, O. et
al(Vinyals et al., 2016) and Santoro, A. et al(Santoro et al.,
2016) which have different split about background set and
evaluation set and different test trails from BPL Test Protocol(Lake et al., 2015b). Vinyals’, O. randomly picked
up 1200 characters from Omniglots background set and
evaluation set as the training set and 423 characters as test
set. In Vinyals split, the training set and test set have iden-

We carried out 5-way and 20-way one-shot and 5-shot classifications, and the comparisons with published results (as
baselines) are shown in Table 2. The models labeled by 60,
156 and 1200 was trained on the test protocol provided by
Vinyals, O. , with 60, 156, 1200 training characters and with
423 test characters, in which the training characters and test
characters belong to same alphabets. The models labeled by
964, small2, tiny2 was trained on the test protocol provided

Local Contrast Learning

by Lake respectively with 964,156,60 training characters
and with 659 test characters, in which the training characters
and test characters belong to completely different alphabets.
It must be noted that the experiments in Table 2 are different
from the experiments in Table 1 in training set, test set and
test trials, so the results in Table 2 cannot match the results
in Table 1.
LCNN outperformed prior approaches just using 156 characters, furthermore, and the results using 60 characters is
comparable to published results, which used 1200 characters or 964 characters. Comparing the results using 156
characters and the results using 1200 or 964 characters,
the latter is not obviously higher than the former, which
demonstrate, for training LCNN on Omniglot, 156 character
is almost enough, and more data cannot very improve the
performance. That proves that LCNN can learn enough
knowledge from small data for discriminating objects. Conversely, more training data might imported more noises that
confused the models and declined the performance, for instance, the 5-way one-shot accuracy 98.97% of the model
LCNN 964 is slightly lower than the accuracy 99.24% of
the model LCNN 156.
Comparing the models LCNN tiny2, LCNN small2, LCNN
964, and models LCNN 60, LCNN 156, LCNN 1200, the
accuracies of former is lower than latter. That proves that
it is more challenging to recognize the images in the novel
alphabets than in novel characters in identical alphabets.
4.3. HWDB
Like the test on Omniglot, we evaluated LCNN on CASIAHWDB1.1(Liu et al., 2011; 2013) that is a Chinese handwritten character dataset for machine learning and is more diversified and confusing than Omniglot. CASIA-HWDB1.1
is a widely used dataset that includes 3755 character classes
and 300 images per class written by 300 drawers. Therefore, in order to more extensively evaluate the performance
and provide a benchmark for other one-shot classification
approaches on the handwritten Chinese characters, we defined a one-shot classification of the handwritten Chinese
characters. CASIA-HWDB1.1 includes training subset with
240 images per character and test subset with 60 images per
character. However we only use first 20 images per character in the training subset for evaluating the performance of
one-shot classification.
We pick up the last 600 characters as test set in 3755 characters (The characters are ordered by GB2312 code), and respectively pick up the first 60, 200, 3155 characters as three
different training set: HWDB60, HWDB200, HWDB3155.
All images are resized to 64 x 64 and not augmented. In all
experiments in this section, the test trials are created using
Algorithm 1 on the test set, and the number of test trials

ntrial is computed by the equation in the previous section.
For comparing the performance on HWDB with other
neural networks approaches, we selected Matching Networks(Vinyals et al., 2016) as baseline. The setup of Matching Networks is same with the setup in the previous section,
however the maximum training steps is 315,500.
We carried out 20-way one-shot and 5-shot classifications.
The results comparing the baselines to LCNN are shown in
Table 3. In all experiments, LCNN outperforms the baseline. LCNN with 60 characters achieved accuracy 89.85%
that outperformed the accuracy 85.83% of Matching Networks with 3155 characters. That demonstrates LCNN can
achieve high performance using small data. Comparing the
results of LCNN HWDB200 to LCNN HWDB3155, we can
find, LCNN HWDB200 achieved almost same one-shot and
5-shot classification accuracy only using 6 percent of the
samples of LCNN HWDB3155. That again demonstrates
that LCNN can learn enough knowledge from small data for
discriminating objects.

5. Discussion
Human can identify a novel object by seeing only a few
samples per class. That can be explained by the principles
of compositionality, causality, learning to learn(Lake et al.,
2015b) and others, and the all principles mainly base on
logical reasoning, however before logically thinking, human
recognizes objects by intuition based on innate abilities and
experiences. Contrast is a kind of innate ability and also is a
kind of way enriching experiences. Contrast capability also
can be advanced by learning. LCL simulates the capability
for recognization.
The success of LCL is partially attributed to the extremely
large quantity of LCC and the three randomness: randomly
selecting classes, randomly selecting samples, randomly
shuffling contrastive objects. They all enforce LCNN to
learn to distinguish objects instead of representing and remembering the training object or its pattern. Learning from
contrasting in local context is another crucial factor. Contrasting in local context enforce LCNN to adapt different
local context, thus LCNN cannot remember the context information, so, in case of a tiny samples, LCNN would not
encounter overfitting. Of course, super deep neural network,
such as Resnet, is very important factor, and the its depth
may provide enough flexibility for adapting tremendous local context. However, LCL assure the super deep neural
network to avoid overfitting in case of tiny samples. Therefore, the super deep neural network contributes to the high
classification accuracy, but LCL contributes to its success
in training. It is fine design to use Difference Perceptron
to contrast the difference embedding instead of learning a
similarity metric. That makes the contrast objects in LCC

Local Contrast Learning
Table 2. One-shot and few-shot Accuracy (%) on Omniglot Based on Variant Test Protocol. Results are accuracies averaged over 100
times runs with 95% confidence intervals where reported. -: not reported.

M ODEL

C HARS

P IXELS N EAREST N EIGHBOR (V INYALS ET AL ., 2016)
MANN(S ANTORO ET AL ., 2016)
M ATCHING N ETS 10
N EURAL S TATISTICIAN (E DWARDS & S TORKEY , 2017)
C ONV N ET WITH M EMORY M ODULE
(K AISER ET AL ., 2017)
P ROTOTYPICAL N ETWORKS (S NELL ET AL ., 2017)
MAML(F INN ET AL ., 2017)
M ETA N ET (M UNKHDALAI & Y U , 2017) 1200
LCNN 60
LCNN 156
LCNN 1200
M ETA N ET (M UNKHDALAI & Y U , 2017) 964
LCNN TINY 2
LCNN SMALL 2
LCNN 964

M ATCHING N ETS
HWDB60
HWDB200
HWDB3155
LCNN
HWDB60
HWDB200
HWDB3155

C HARS

1- SHOT

5- SHOT

60
200
3155

69.78
81.67
85.83

-

60
200
3155

89.85±0.26
96.33±0.13
97.32±0.16

97.70±0.13
99.41±0.06
99.60±0.06

20- WAY ACC .
1- SHOT
5- SHOT

1200
1200
1200
1200

41.7
82.8
98.1
98.1

94.9
98.9
99.5

93.8
93.2

98.7
98.1

1200
1200
1200
1200
60
156
1200
964
60
156
964

98.4
98.8
98.7±0.4
98.95
98.95±0.06
99.24±0.05
98.97±0.05
98.45
95.36±0.09
98.54±0.06
98.86±0.09

99.6
99.7
99.9±0.1
99.70±0.03
99.77±0.03
99.76±0.03
99.04±0.05
99.63±0.04
99.70±0.07

95
96
95.8±0.3
97
96.84±0.16
98.03±0.13
98.28±0.12
95.92
95.88±0.20
97.01±0.13
97.77±0.22

98.6
98.9
98.9±0.2
99.05±0.10
99.31±0.09
99.46±0.07
98.85±0.11
99.15±0.08
99.33±0.18

Table 3. 20-way Classification Accuracy (%) on HWDB. Results
are accuracies averaged over 100 times runs and with 95% confidence intervals where reported.The models labeled by HWDB60,
HWDB200, HWDB3155 was respectively trained on first 60, 200,
3155 characters of HWDB with 20 samples per character.

M ODEL

5- WAY ACC .
1- SHOT
5- SHOT

26.7

instance, in Table1, the accuracy 98.92% of 964 characters
is slightly less than the accuracy 99.17% of 156 characters,
because in Omniglot there are some similar characters in
different alphabets, that is noise to LCNN.
LCL can be a fundamental tool for setting up artificial intelligence systems. More high level architectures can be invented based on LCL. For example, the hierarchical LCNN
could be used to distinguish hierarchical classes. If the objects in LCC are sequential, the sequential LCNN could be
created.
The theoretical explanations for LCL from perspective of
cognitive neuroscience and cognitive psychology are expected to study by the researchers in those fields.

6. Conclusion
become a list rather than a set, and makes the order of the
contrast objects become meaningful. The order is crucial to
create tremendous different LCC.
The number of training classes and the number of samples
per class are an essential factor to improve the performance.
More classes and samples per class can improve the test
accuracy. However, the accuracy is only increased when
the sample is less. In case the training sample is enough,
the increase of sample cannot almost improve the classification accuracy, which is similar with human beings cognitive
behavior that human can learn to recognize object by few
samples, but human cannot almost learn more from redundant similar samples. In some cases, the increase of sample
might worsen the accuracy, because more samples maybe
bring in more noise which can interfere the training. For

We proposed a novel deep learning approach named Local
Contrast Learning (LCL) for alleviating deep models overfitting resulted by lack for training samples. LCL enforce the
deep model to adapt tremendous local contexts and carefully
capture difference between contrastive object and recognizing object. LCL is able to stably and successfully train deep
neural network with more than100 layers using dozens of
sample classes and tens of samples each class. The approach
achieves state-of-the-art results using 6 percent samples of
baselines. The results proves that the deep model can be
trained using very small data.

Local Contrast Learning

References
Bell, Sean and Bala, Kavita. Learning visual similarity
for product design with convolutional neural networks.
ACM Transactions on Graphics, 34(4):98, 2015. ISSN
07300301. doi: 10.1145/2766959.
Edwards, Harrison and Storkey, Amos. Towards a neural
statistician. In ICLR, 2017.
Fei-Fei, Li, Fergus, Rob, and Perona, Pietro. One-shot learning of object categories. IEEE transactions on pattern
analysis and machine intelligence, 28(4):594–611, 2006.
Finn, Chelsea, Abbeel, Pieter, and Levine, Sergey. Modelagnostic meta-learning for fast adaptation of deep networks. In ICML, 2017.
Grm, K., Dobrisek, S., and Struc, V. Deep pair-wise similarity learning for face recognition. In 2016 4th International
Conference on Biometrics and Forensics (IWBF), pp. 1–6,
2016. doi: 10.1109/IWBF.2016.7449690.
Hadsell, R., Chopra, S., and LeCun, Y. Dimensionality
reduction by learning an invariant mapping. In CVPR,
volume 2, pp. 1735–1742, 2006. doi: 10.1109/CVPR.
2006.100.
He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun,
Jian. Delving deep into rectifiers: Surpassing human-level
performance on ImageNet classification. In Proceedings
of the IEEE international conference on computer vision,
pp. 1026–1034.
He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun,
Jian. Deep residual learning for image recognition. In
Proceedings of the IEEE conference on computer vision
and pattern recognition, pp. 770–778, 2016a.
He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun,
Jian. Identity mappings in deep residual networks.
arXiv:1603.05027 [cs], 2016b.
Hoffer, Elad and Ailon, Nir. Deep metric learning using
triplet network. In ICLR, 2015.
Ioffe, Sergey and Szegedy, Christian. Batch normalization:
Accelerating deep network training by reducing internal
covariate shift. In Proceeding ICML’15, volume 37, pp.
448–456, 2015.
Kaiser, ukasz, Nachum, Ofir, Roy, Aurko, and Bengio, Samy.
Learning to remember rare events. In ICLR, 2017.
Koch, Gregory, Richard Zemel, and Ruslan Salakhutdinov.
Siamese neural networks for one-shot image recognition.
2015.

Kumar, B. G., Carneiro, Gustavo, Reid, Ian, and others.
Learning local image descriptors with deep siamese and
triplet convolutional networks by minimising global loss
functions. In CVPR, pp. 5385–5394, 2016.
Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B.
Human-level concept learning through probabilistic program induction supplementary material. Science, 350
(6266):1332–1338, 2015a. ISSN 0036-8075, 1095-9203.
doi: 10.1126/science.aab3050.
Lake, Brenden M., Salakhutdinov, Ruslan R., and Tenenbaum, Josh. One-shot learning by inverting a compositional causal process. In Advances in neural information
processing systems, pp. 2526–2534, 2013.
Lake, Brenden M., Salakhutdinov, Ruslan, and Tenenbaum,
Joshua B. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–
1338, 2015b.
LeCun, Yann, Bengio, Yoshua, and Hinton, Geoffrey. Deep
learning. Nature, 521(7553):436–444, 2015. ISSN 00280836, 1476-4687. doi: 10.1038/nature14539.
Lin, Min, Chen, Qiang, and Yan, Shuicheng. Network in
network. arXiv preprint arXiv:1312.4400, 2013.
Liu, Cheng-Lin, Yin, Fei, Wang, Da-Han, and Wang, QiuFeng. CASIA online and offline chinese handwriting
databases. In ICDAR, pp. 37–41, 2011. ISBN 978-14577-1350-7. doi: 10.1109/ICDAR.2011.17.
Liu, Cheng-Lin, Yin, Fei, Wang, Da-Han, and Wang, QiuFeng. Online and offline handwritten chinese character
recognition: Benchmarking on new databases. Pattern
Recognition, 46(1):155–162, 2013. ISSN 0031-3203. doi:
10.1016/j.patcog.2012.06.021.
Miller, George A. The magical number seven, plus-or-minus
two, some limits to our capacity for processing information. Brain Physiology and Psychology. Buttenvorths:
London, pp. 175–200, 1967.
Munkhdalai, Tsendsuren and Yu, Hong. Meta networks. In
ICML, 2017.
Rumelhart, David E., Hinton, Geoffrey E., and Williams,
Ronald J. Learning representations by back-propagating
errors. Nature, 323(6088):533–536. ISSN 0028-0836.
doi: 10.1038/323533a0.
Santoro, Adam, Bartunov, Sergey, Botvinick, Matthew,
Wierstra, Daan, and Lillicrap, Timothy. Meta-learning
with memory-augmented neural networks. In International conference on machine learning, pp. 1842–1850,
2016.

Local Contrast Learning

Snell, Jake, Swersky, Kevin, and Zemel, Richard S. Prototypical networks for few-shot learning. arXiv:1703.05175
[cs, stat], 2017.
Sutskever, Ilya, Martens, James, Dahl, George, and Hinton, Geoffrey. On the importance of initialization and
momentum in deep learning. In ICML, pp. 1139–1147.
Vinyals, Oriol, Blundell, Charles, Lillicrap, Tim, Wierstra,
Daan, and others. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, pp. 3630–3638, 2016.
Wang, Jiang, Song, Yang, Leung, Thomas, Rosenberg,
Chuck, Wang, Jingbin, Philbin, James, Chen, Bo, and
Wu, Ying. Learning fine-grained image similarity with
deep ranking. In CVPR, pp. 1386–1393, 2014.

