Few-shot Learning by Exploiting Visual Concepts within CNNs

arXiv:1711.08277v3 [cs.CV] 13 Feb 2018

Boyang Deng1∗ , Qing Liu2 , Siyuan Qiao2 , Alan Yuille2
1
Beihang University
2
Johns Hopkins University
billydeng@buaa.edu.cn, {qingliu, siyuan.qiao, alan.yuille}@jhu.edu

Abstract
Convolutional neural networks (CNNs) are one of
the driving forces for the advancement of computer
vision. Despite their promising performances on
many tasks, CNNs still face major obstacles on the
road to achieving ideal machine intelligence. One
is that CNNs are complex and hard to interpret. Another is that standard CNNs require large amounts
of annotated data, which is sometimes hard to obtain, and it is desirable to learn to recognize objects
from few examples. In this work, we address these
limitations of CNNs by developing novel, flexible,
and interpretable models for few-shot learning. Our
models are based on the idea of encoding objects
in terms of visual concepts (VCs), which are interpretable visual cues represented by the feature
vectors within CNNs. We first adapt the learning
of VCs to the few-shot setting, and then uncover
two key properties of feature encoding using VCs,
which we call category sensitivity and spatial pattern. Motivated by these properties, we present two
intuitive models for the problem of few-shot learning. Experiments show that our models achieve
competitive performances, while being more flexible and interpretable than alternative state-of-theart few-shot learning methods. We conclude that
using VCs helps expose the natural capability of
CNNs for few-shot learning.

1

Introduction

After their debut [LeCun et al., 1998], Convolutional Neural Networks (CNNs) have played an ever increasing role in
computer vision, Some researchers have even claimed that
CNNs have surpassed human-level performance [He et al.,
2015], although other work suggests otherwise [Zhu et al.,
2017]. Recent studies also show that CNNs are vulnerable
to adversarial attacks [Goodfellow et al., 2015]. Nevertheless, the successes of CNNs have inspired the computer vision community to develop increasingly sophisticated models [Szegedy et al., 2017].
∗

Work done during visiting Johns Hopkins University

Despite the impressive achievements of CNNs, we have
limited insights into why CNNs are effective. The everincreasing depth and complicated structures of CNNs make
them difficult to interpret while the non-linear nature of
CNNs makes it hard to perform theoretical analysis. In addition, CNNs require large annotated datasets which is problematic for many real world applications. We argue that the
ability to learn from a few examples, or few-shot learning, is a
characteristic of human intelligence and is strongly desirable
for an ideal machine learning system.
The goal of this paper is to develop an approach to interpretable and flexible few-shot learning which builds on the
successes of CNNs. We start from the intuition that objects
can be represented in terms of spatial patterns of parts which
implies that new objects can be learned from a few examples
if they are built from parts that are already known, or which
can be learned from a few examples. We recall that previous
researchers have argued that object parts are represented by
the convolutional layers of CNNs [Zhou et al., 2015; Mahendran and Vedaldi, 2015] provided the CNNs are trained for
object detection. More specifically, we will build on recent
work [Wang et al., 2015] which learns a dictionary of Visual
Concepts (VCs) from CNNs representing object parts, see
Figure 1. Their original work proves that these VCs can be
combined to detect semantic parts. More recently, it has also
been shown that VCs can be used to represent objects using VC-Encoding (where objects are represented by binary
codes of VCs).
But it is not obvious that VCs, as described in [Wang et al.,
2017], can be applied to few-shot learning. Firstly, these VCs
were learned independently for each object category (e.g., for
cars or for airplanes) using deep network features from CNNs
which had already been trained on these categories. Secondly,
the VCs were learned using large numbers of examples of
the object category, ranging from hundreds to thousands. By
contrast, for few-shot learning we have to learn the VCs from
a much smaller number of examples (by an order of magnitude or more). Moreover, we can only use deep network
features which were trained on datasets not including those
object categories that we hope to learn within few shots. This
means that although we will extract VCs using very similar
algorithms to those in [Wang et al., 2015] our motivation and
problem domain are very different. To summarize, in this paper we use VCs to learn models of new object categories from

(a) VC 139
(“Sofa Cushion”)

(b) VC 189
(“Side Windows”)

(c) VC 174
(“Bicycle Wheel”)

Figure 1: Visualizations of VCs. Each group consists of patches
from original images closest to a VC. In general, these patches
roughly correspond to semantic parts of objects, e.g., the cushion
of a sofa (a), the side windows of trains (b) and the wheels of bicycles (c). All VCs are referred to by their indices (e.g., VC 139).
We stress that VCs are learned in an unsupervised manner and
terms like“sofa cushion” are inferred by observing the closest image patches and are used to describe them informally.

existing models of other categories, while [Wang et al., 2015]
uses VCs to help understand CNNs and to perform unsupervised part detection. Also, unlike [Wang et al., 2015], we use
VC-Encoding.
In Section 3, we will review VCs in detail. Briefly speaking, VCs are extracted by clustering intermediate-level features of CNNs, e.g., features produced by the Pool-4 layer
of VGG16 [Simonyan and Zisserman, 2015]. Serving as the
cluster centers in feature space, VCs divide intermediate-level
features into a discrete dictionary. We show that VCs can be
learned in the few-shot learning setting and they have two desirable properties when used for image encoding, which we
call category sensitivity and spatial pattern.
More specifically, we develop an approach to few-shot
learning which is flexible and interpretable. We learn a dictionary of VCs as described above which enables us to represent
novel objects by their VC-Encoding. Then we propose two
intuitive models: (i) nearest neighbor and (ii) a factorizable
likelihood model based on the VC-Encoding. The nearest
neighbor model uses a similarity measure to capture the difference between two VC-Encodings. The factorizable likelihood model learns a likelihood function of the VC-Encoding
which, by assuming spatial independence, can be learned
from few examples. We emphasize that both these models
are very flexible, in the sense that they can be applied directly
to any few-shot learning scenarios. This differs from other
approaches which are trained specifically for scenarios such
as 5-way 5-shot (where there are 5 object categories with 5
examples of each). This flexibility is attractive for real world
applications where the numbers of new object categories, and
the number of examples of each category, will be variable.
Despite their simplicity, these models achieve comparable results to the state-of-the-art few-shot learning methods, such
as learning a metric and learning to learn. From a deeper
perspective, our results show that CNNs have the potential
for few-shot learning on novel categories but to achieve this
potential required studies of the internal structures of CNNs
to re-express them in simpler and more interpretable terms.
Overall, our major contributions are two-fold:
(1) We show that VCs can be learned in the few-shot set-

ting using CNNs trained on other object categories. By
encoding images using VCs, we observe two desirable
properties, i.e., category sensitivity and spatial pattern.
(2) Based on these properties, we present two simple, interpretable, and flexible models for few-shot learning.
These models yield competitive results compared to the
state-of-the-art methods on specific few-shot learning
tasks and can also be applied directly, without additional
training, to other few-shot scenarios.

2

Related Work

Our work on few-shot learning is motivated by and builds on
attempts to understand the internal representations of neural
networks. Therefore, we review here the previous literature
on these topics.

2.1

Neural Network Internal Representations

Recently, there have been numerous studies aimed at understanding the behavior of neural networks and, in particular,
to uncover the internal representations within CNNs. Some
try to visualize internal representations by sampling [Zeiler
and Fergus, 2014], generating [Simonyan et al., 2013] or by
backpropagating [Mahendran and Vedaldi, 2015] images in
order to maximize the activations of the hidden units. A
particularly relevant work by [Zhou et al., 2015] shows that
object and object parts detectors emerge in CNNs. Conversely, other works investigate the discriminative power of
the hidden features of CNNs by assessing them on specific
problems [Sharif Razavian et al., 2014; Bau et al., 2017;
Yosinski et al., 2014]. The overall findings suggest that deep
networks have internal representations of object parts. The
most relevant work to our paper is the study of VCs which
discovered mid-level visual cues in the internal features of
CNNs and showed relationships between these visual cues
and semantic parts [Wang et al., 2015; Wang et al., 2017].
This work is described in detail in Section 3

2.2

Few-Shot Learning

There have been growing attempts to perform few-shot learning motivated by attempts to mimic human abilities and to
avoid some of the limitations of conventional data-demanding
learning. An early attempt was made building on probabilistic program induction [Lake et al., 2015]. More recent
efforts at few-shot learning can be broadly categorized into
two classes. The first is to design methods to embed the inputs into a feature space friendly to few-shot settings [Koch
et al., 2015; Vinyals et al., 2016]. Their goal is to find
a good similarity measure (e.g., using Siamese networks)
that can be applied rapidly to novel categories. The second
is meta-learning which efficiently trains an ordinary model
with the budget of few examples [Ravi and Larochelle, 2017;
Finn et al., 2017]. An alternative approach by [Qiao et
al., 2017] performs few-shot learning by estimating parameters of the prediction layer using regression from previously
learned objects. We emphasize that the approach in our paper
differs from these works, many of which are tailored for a few
specific few-shot learning scenarios (i.e., test and train conditions must match), while our methods are simple and flexible,
so they work both in normal and almost all few-shot settings.

Figure 2: Key terms in the VC formalization. On the left is the n-th
input image, defined on image lattice Ln
0 , with height H0 and width
W0 . In the middle is the lattice at the kth layer of the CNN for
the n-th image, noted by Ln
k , with height Hk and width Wk . On
n
the right is a feature vector at position p in Ln
k , noted by fp , with
dimensionality Ck .

feature vector we get from the intermediate layer of a CNN
(without image identity or spatial information). α, µ, and κ
are vMFM parameters and represent the mixing proportion,
mean direction, and concentration values respectively.
We define the set of VCs by V = {µv : v = 1, . . . , V } (i.e.,
by the mean directions of the learned vMFM distribution).
Alternatively, since the {µv } have the same dimensionality
as the {fm }, we denote a specific VC center by fv = µv .
To help understand the VCs, we compute the cosine distances from the original feature vectors to the VCs as follows:
dnp,v = 1 −

3

Background: Visual Concepts

In [Wang et al., 2015], VCs were discovered as internal representations within deep networks which roughly correspond
to mid-level semantic visual cues. These VCs play a core role
in our work on understanding properties of CNNs and developing our interpretable few-shot learning models. In this section, we review how VCs were defined and learned in [Wang
et al., 2015]. We will describe later in Section 4.1 how we
modify VCs for few-shot learning.
We first summarize the formalization of VCs, which are illustrated in Figure 2. CNNs contain a hierarchy of lattices Ll ,
where l ∈ {0, 1, . . .} stands for the layer of the lattice. In particular, the input image is defined over the lattice L0 and the
lattice on which we derive VCs is specified by Lk . We denote
the spatial mappings from L0 to Lk by π07→k and from Lk to
L0 by πk7→0 . Then we define Fkn = {fpn : p ∈ Lnk } as the
feature vector set at Lnk of the n-th image where p refers to a
2D position in the lattice Lnk . These feature vectors are computed by fpn = f (InA(p0 ) ), where the function f is specified
by the neural network and InA(p0 ) is a subregion of the n-th
input image In , centered on a point p0 = πk7→0 (p) on Ln0 . In
other words, the responses of all the channels in position p
constitute the feature vector fpn . Then we have Fk = ∪n Fkn
from all images of interest. Note that by collecting feature
vectors into Fk , all spatial and image identity information
is removed. Since layer k is usually pre-selected for different network architectures for VCs applications (e.g., [Wang
et al., 2015] typically studied layer k = 4 for VGG16 net in
their work), the subscript will be omitted in the remainder of
the paper for simplicity.
Now we describe how VCs are extracted. The approach
assumes that the VCs are represented by a population code
of the CNN feature vectors. They are extracted using an unsupervised clustering algorithm. Since we first normalize the
feature vectors into unit length, instead of using K-means as
proposed in [Wang et al., 2015], we assume that the feature
vectors are generated by a mixture of von Mises-Fisher distributions (vMFM) and learn this mixture by the EM algorithm [Banerjee et al., 2005]. The goal is to maximize the
likelihood function
V
P (F|α, µ, κ) = ΠM
m=1 Σv=1 αm,v Vd (fm |µv , κv ),

(1)

where M is the total feature vector count (i.e., the number
of all feature vectors collected from all images of interest)
and V is the predefined VC (cluster) number. Vd (·) is the
density function of the vMF distribution. fm denotes each

fpn · fv
fpn

2

kfv k2

,

(2)

where dnp,v denotes the distance between feature vector fpn
and the VC v in the n-th image at position p, and we call them
VC distances. We select those feature vectors with the smallest distances to each VC and trace them back to the original
input image using πk7→0 . This yields “visualization patches”
of VCs, shown in Figure 1. We observe that these patches
roughly correspond to the semantic parts of objects, which
justifies our assertion that VCs are semantic visual cues.
In the previous studies of VCs [Wang et al., 2015; Wang et
al., 2017], the CNNs that were used to generate feature vectors were trained for a large scale object classification task
that included the object categories of interest. Moreover,
they extracted VCs using hundreds of images within a specific category of object, which resulted to category specific
visual cues that were useful for interpreting CNN behaviors
and building novel models for semantic part detections. In
more recent work (in preparation) VCs were used to encode
semantic parts and objects using VC-Encoding that could be
applied to detection tasks in the presence of occlusion. VCEncoding is described in the next section. We emphasize that
none of this prior work on VCs addressed few-shot learning
and only addressed situations where there were many training
examples of the object categories.

4

Few-Shot Learning from VCs

This section describes the technical ideas of our paper. In
Section 4.1, we introduce how we learn VCs in the few-shot
setting. In Section 4.2, we introduce VC-Encoding and show
its two desirable properties for few-shot classification tasks.
Then in Section 4.3 and Section 4.4, we propose two simple
and interpretable VC-Encoding models for few-shot learning.

4.1

Few-shot VCs

It is not obvious that VCs can be applied to few-shot learning
tasks where only few examples are available for each novel
category. It is not possible to train the CNNs on all the object
categories (as was done in [Wang et al., 2015]) and also there
may not be enough data to get good VC clusters. Hence we
modify the way VCs are learned: we learn VCs from small
number of examples of novel object categories using features
from CNNs trained on other object categories. This is similar to how metric-learning and meta-learning are trained on
large datasets which do not include the novel categories and
ensures that the CNN used for feature extraction has never
seen the categories on which we will perform the few-shot

Visual
Concept

Input
Image

Visualize

VC_2

VC_3

9.2%
2.4%
24.5%
40.5%
13.2%
10.3%

11.5%
9.4%
12%
11.8%
45.2%
10.2%

VC_42:

VC_9

Index
Airplane
Bicycle

Airplane

Bus

Bicycle

CarBus
Motorbike
Car
Train
Motorbike

Train

77.1%
VC_1
7.5%

77.1%
7.5%
5%
0.5%
3%3%
6.9%3%
6.9%
0.5%

VC_71
9.2%
VC_2
2.4%

9.2%
2.4%
40.5%
24.5%
13.2%
40.5%
10.2%
13.2%
10.3%
24.5%

VC_196

Negative
Distance
Map

11.4%
VC_3
9.4%

11.5%
9.4%
12%
45.2%
11.8%
10.2%
45.2%
10.2%
12%

11.8%

Binary
Map

(a) Category Sensitivity

(b) Spatial Pattern

Figure 3: Properties of VCs. In (a), we illustrate three VCs by their
closest patches and their occurrence distributions over 6 object categories out of the 12 in PASCAL3D+ showing category sensitivity
of VC-Encoding. In (b), we visualize the closest patches to VC 42
in the top green box and randomly select 4 images of cars with their
negative distance maps and binary maps w.r.t. VC 42 plotted in the
bottom orange box. The negative distance map is given by −dp,42
and is scaled to (0, 1). The binary map is drawn based on bp,42 . See
Section 4.2 for more details.

task. To extract VCs for the novel categories which only have
few examples each, we pool feature vectors from the different categories together and perform the clustering algorithm
on all of them. This gives us a little more data and encourages
VC sharing between different categories, which also makes it
easier to apply our VC models to multiple novel categories.
By the two modifications described above, we obtain fewshot VCs, i.e., VCs that are suitable for few-shot learning.
This is critical for our application and differentiates this work
from previous studies of VCs (e.g., [Wang et al., 2015]). Surprisingly, we find that we only need a few images (e.g., five
images per category) to extract high quality VCs (see visualizations in Figure 3a) which, when used for VC-Encoding,
possess similar desirable properties as the traditional VCs and
hence are suitable for few-shot object classification task.

4.2

threshold ensuring that the average coverage >= 0.8 for all
few-shot training images. This yields the final VC-Encoding
B used in our models, with the following desirable properties:
Category Sensitivity Despite the fact that the VCs are
learned from a mix of images with different category labels,
the first insight is that many VCs tend to fire (b·,v = 1) intensively for one or a few specific object categories. In Figure 3a,
we calculate the occurrence distributions of several VCs for
6 object categories out of the 12 in PASCAL3D+ [Xiang et
al., 2014]. In each column that represents a specific VC, the
occurrence frequencies tend to be high for one or two object
categories and low for the others. This suggests that VC identities can provide useful informations for object classification.
Moreover, the corresponding visualized patches on the top of
Figure 3a support our understanding that VCs have this category sensitivity because they capture the semantic parts that
are specific for object categories.
Spatial Pattern The spatial pattern of VC firings is also
indicative of the object category. Although spatial information is ignored during feature clustering, the learned VCs give
binary maps that contain regular spatial patterns for images
of the same category with relatively similar viewpoints (as
shown in Figure 3b). This is consistent with the more general conjecture that the spatial patterns of semantic parts play
a vital role in object recognition, and shows again that the
VC-Encoding can capture the spatial patterns of the semantic
parts to a certain extend.
Next, we design two simple few-shot learning models
based on VC-Encoding learned from few examples.

4.3

Nearest Neighbor on Spatial Patterns

First, we propose a simple template matching model which is
similar to traditional nearest neighbor algorithms. The novelty is that we use a similarity metric between VC-Encodings
which is spatially “fuzzy” so that it can tolerate small spatial
shifts of the parts in images. Formally, the similarity metric
takes the following form:

VC-Encoding

We assume that objects can be decomposed into semantic
parts. From the perspective of VCs, this means that most
{fp } should be assigned to a single VC. This requires specifying an explicit relationships between the {fp } and the VCs.
A natural choice is to compute the distances dp,v between
the fp and the v-th VC and threshold it to produce a binary value bp,v (i.e., bp,v = 1 if dp,v < T ). We refer to
B = {bp,v : p ∈ L, v = 1, . . . , V } as the VC-Encoding.
Note the image index n is omitted here since the operations
are identical for all images of interest. We use two criteria to
specify a good encoding, coverage and f irerate, defined as
following:
coverage

=

f irerate

=

P

maxv bp,v

P

|L|
P

p∈L

p∈L

|L|

v

bp,v

.

,

(3)
(4)

The choice of the encoding threshold T is a trade-off between requiring sufficient coverage and a firing rate that is
close to one. In practice, we choose T for each testing trial by
a grid-search with step size 0.001 which outputs the smallest

1
K(b, b ) = (
2
0

P

p,v

bp,v maxq,q∈n(p) b0q,v
P
+
p,v bp,v

P

p,v

b0p,v maxq,q∈n(p) bq,v
P
),
0
p,v bp,v
(5)

where K(b, b0 ) is the similarity between the binary VCEncodings b and b0 . n(p) defines the set of neighboring
Position P
Neighbor Positions of P
positions of p (as shown in
Figure 4
Figure 4). During testing, we
classify an image to the category of the training example with
the largest similarity.
P

4.4

P

Factorizable Likelihood Model

Apart from the intuitive nearest neighbor method, we present
a second method which models the likelihood of the VCEncoding. We observe that we can specify a distribution
over the VC-Encoding bp,v using a bernoulli distribution with
probability θp,v . Following Naı̈ve Bayes, we assume all the
elements of the VC-Encoding b are independent (making it
possible to learn the distribution from a very small number
of examples). Hence we can express the likelihood of b as

following:
L(b|θ) =

Method
Y

bp,v · θp,v + (1 − bp,v ) · (1 − θp,v ).

(6)

p,v

For each object category y, we derive a probabilistic distribution θy from the training examples. Thus the prediction of
object category given the VC-Encoding b is given by:
yb = max L(b|θy ).
y

(7)

Note that by doing this, we are in fact implementing a discriminative model obtained from a generative distribution.
We smooth each distribution θy using a Gaussian filter to
guard against unlikely events.

5

Experiments

Section 4.1 suggests that a few images may be enough for
learning object models when represented by VC-Encodings.
Indeed our experiments show that both our two VCs-based
few-shot learning models are competitive in performance
with alternative methods designed specifically for few-shot
learning such as [Ravi and Larochelle, 2017]. In addition,
while previous few-shot methods are trained to work in specific few-shot scenarios, such as 5-way classifications, our
methods can be applied to a large range of few-shot scenarios
without additional training. The experimental results show
that trained CNNs have the potential to recognize novel objects from few examples by exploiting VC-Encoding.

5.1

Mini-ImageNet

To assess the capability of our few-shot methods, we first
evaluate them on a common few-shot learning benchmark,
namely Mini-ImageNet. The Mini-ImageNet dataset was first
proposed by [Vinyals et al., 2016] as a benchmark for evaluating few-shot learning methods. It selects 100 categories
out of 1000 categories in ImageNet with 600 examples per
category. We use the split proposed by [Ravi and Larochelle,
2017] consisting of 64 training categories, 16 validation categories and 20 testing categories. In accordance with the convention for Mini-ImageNet, we perform numerous trials of
few-shot learning during testing. In each trial, we randomly
sample 5 unseen categories from a preserved testing set. Each
category is composed of 5 training images for the 5-shot setting and 1 training image for the 1-shot setting. During evaluation, we randomly select 15 images for each category following [Ravi and Larochelle, 2017].
For our methods, we train a VGG-13 on the training and
validation set with the objective of cross entropy. We preserve 10% images in each category to validate our network.
Then, we extract 200 VCs from the Pool-3 layer for the current testing pool. The reason for choosing Pool-3 features is
that a grid in Pool-3 lattices L3 correspond to a 36 × 36 patch
in the original 84 × 84 image, which is a plausible size for
a semantic part. For the Gaussian filter used to smooth the
factorizable likelihood model, we use σ = 1.2.
As Table 1 illustrates, we compare our methods against
two baselines in line with the ones in [Ravi and Larochelle,
2017] (referred to by Baseline-FT and Baseline-NN). To directly examine the impact of the VCs, we also include the

5-category
1-shot

5-shot

Baseline-FT
Baseline-NN
Pool3-NN
MatchingNet
Meta-Learner
MAML

28.86 ± 0.54%
41.08 ± 0.70%
43.38 ± 0.81%
43.56 ± 0.84%
43.44 ± 0.77%
48.70 ± 1.84%

49.79 ± 0.79%
51.04 ± 0.65%
55.33 ± 0.75%
55.31 ± 0.73%
60.60 ± 0.71%
63.11 ± 0.92%

VC-NN (Ours)
VC-LH (Ours)

46.39 ± 1.09%
45.61 ± 1.14%

58.84 ± 1.12%
63.07 ± 1.02%

Table 1: Average classification accuracies on Mini-ImageNet with
95% confidence intervals. Evaluations of Baseline-FT(finetune)
and Baseline-NN(nearest neighbor) are from [Ravi and Larochelle,
2017]. Pool3-NN stands for a nearest neighbor method based on
raw Pool-3 features from the same VGG-13 as our methods. At
the bottom are our nearest neighbor method (VC-NN) and factorizable likelihood method (VC-LH) based on VCs. Marked in bold
at the top are the best published results for each scenario. Marked
in bold at the bottom are our best results for the corresponding setup. Note we adopt the results for Matching Network from [Ravi and
Larochelle, 2017].

result of nearest neighbor matching using raw features from
the pool-3 layer (referred to by Pool3-NN). In addition, we
present the performances of state-of-the-art few-shot learning methods, including MatchingNet [Vinyals et al., 2016],
Meta-Learner [Ravi and Larochelle, 2017], and MAML [Finn
et al., 2017]. The results show that our VCs-based methods compare well with current methods which were specifically designed for few-shot learning. Compared with metalearning-based methods, we achieve higher accuracy than
Meta-Learner both in the 1-shot and the 5-shot set-ups, while
being just slightly behind MAML. Compared with metricbased methods, which are more similar to ours, we marginally
outperform the MatchingNet.
Moreover, we evaluate out methods’ few-shot learning
ability with variances of the settings (e.g., number of shots
and number of categories) and the results are listed in Table 2.
Here we use exactly the same model as the ones in Table 1
which are trained only once on the training and validation set.
To our knowledge, all of the state-of-the-art few-shot learning methods listed in Table 1 cannot deal with changes in the
6-category
3-shot

8-category
4-shot

Baseline-NN
Pool3-NN

46.70 ± 0.84%
44.25 ± 0.73%

42.48 ± 0.74%
43.30 ± 0.73%

VC-NN (Ours)
VC-LH (Ours)

50.42 ± 0.97%
52.41 ± 0.93%

46.39 ± 0.74%
47.37 ± 0.74%

Method

Table 2: Average classification accuracies on Mini-ImageNet with
95% confidence intervals under randomly selected few-shot settings.
All models used here are the same as the ones used in Table 1. Our
method adapts easily to different number of categories and different
number of shots with minimal re-training and consistently outperform baseline methods, while the other state-of-the-art cannot be
directly applied to these settings.

Low

Universal Color Bar

High

Step I:
Transform input to
Pool-4 Raw Features.

Step III:
Given a set of VC distribution, calculate Pixel-wise Likelihood.

Step IV:
Aggregate Pixel-wise Likelihood
into Class-wise Likelihood.

airplane
Step II:
Transform Raw Features into VC-Encoding.

bicycle

airplane
bicycle

boat

sofa

Raw Features

VC-Encoding

sofa
Distribution

airplane bicycle

boat

train

Step V:
Choose the
class with
highest
likelihood.

Visual Concept

sofa

Decision:
tvmonitor

train
tvmonitor

Class-wise
tvmonitor Likelihood

train tvmonitor

Figure 5: Visualizing the inference procedure of the factorizable
likelihood model (using one VC for example). For better visualization, we rescale the variance of aggregated likelihood to 1. For
all the visualizations, we use the same Universal Color Bar. This
figure is best viewed in color. See Section 5.2 for more details.

number of categories easily. By contrast, our few-shot learning methods based on VC-Encoding can be extended with
minimal re-training to any number of shots and any number
of categories and consistently outperform baseline methods.

5.2

12-category
1-shot
5-shot

–
–

36.12%
32.66%

52.30%
52.46%

VC-LH (Ours)
VC-LH (Ours)
VC-LH (Ours)

120
200
300

39.25%
40.02%
39.23%

64.37%
66.00%
66.47%

VC-NN (Ours)
VC-NN (Ours)
VC-NN (Ours)

120
200
300

40.74%
42.36%
41.18%

58.52%
59.47%
61.07%

Pool4-NN
Pool4-SVM

Pixel-wise
Likelihood

VGG-16

Number of VCs

boat

Pool4
Features

Input

Method

PASCAL3D+

Here we apply our methods to PASCAL3D+ [Xiang et al.,
2014], a dataset with larger high quality images than MiniImageNet. It was originally tailored for 3D object detection and pose estimation by augmenting 12 rigid categories
from PASCAL VOC 2012. Since we interpret our few-shot
recognition mainly by visualizing every step of the inference,
with input images of sufficient sizes, we can obtain large VCEncoding distribution maps whose visualizations are easy for
humans to understand.
On PASCAL3D+, we first give intuition for the interpretability of our model. In Figure 5, we visualize every step
of the inference process using our method based on an example VC. By looking at the closest patches for the VC of
interest, we find this VC is very likely to relate to the corners
of TV Monitors. Then, for a given test image, we convert its
original VGG16 Pool-4 features into VC-Encoding and observe the example VC mainly appears at upper right corner of
the encoding map. After calculating the pixel-wise likelihood
using the distributions learned from a few training images in
each candidate category, it is clearer that except for the TV
Monitor, all other categories show low likelihood in the area
of the corner. Finally, we aggregate the likelihood and make
the correct classification decision by assigning the test image
to the “TV Monitor” category.
Meanwhile, we quantitatively evaluate our methods on
PASCAL3D+. To get a feature extractor CNN, we use a subset of the ImageNet [Deng et al., 2009] classification dataset
which excludes object categories that relate to the 12 categories used in PASCAL3D+ (956 categories left), and we
train an ordinary VGG-16 which achieves 71.27% top-1 accuracy. For testing, we crop the objects out using annotated
bounding boxes provided by PASCAL3D+ and resize them
into 224 × 224. Then we use their Pool-4 features to implement our few-shot methods. As a comparison, we propose
two baseline models. One is a nearest neighbor method based
on raw Pool-4 features using the cosine distance metric; the

Table 3: Average classification accuracies on PASCAL3D+. At the
top are the baseline methods including nearest neighbor (Pool4-NN)
and Exemplar-SVM (Pool4-SVM), based on Pool-4 features from
the same VGG-16 used in our methods. In the middle and the bottom are our factorizable likelihood models (VC-LH) and VCs-based
nearest neighbor models (VC-NN) using different number of VCs
respectively. Marked in bold are the best results within each group
for each scenario.

other is an Exemplar-SVM trained using hinge loss. Both of
them use the same pre-trained VGG-16 as our VC methods.
During evaluation, we set 20 trials of both 5-shot and 1-shot
learning over 12 categories on PASCAL3D+. We also assess
our methods using different numbers of VCs. The results are
shown in Table 3.
In general, our VC-based methods consistently outperform
two baselines by large margins. In particular, the improvements we achieve (especially in our nearest neighbor models) compared to the baseline methods are due to the use of
VCs (e.g., by transferring the raw feature vectors into VC
distances, and by thresholding the distance to get the VCEncoding). Thus, we claim that decomposing fuzzy features
(i.e., deep network features) into explicit semantic cues (i.e.,
the VCs) improves both interpretability and performance. We
also notice that our methods are not sensitive to the number
of VCs since changes of the number of VCs only cause slight
differences among accuracies.

6

Conclusion

In this paper we address the challenge of developing simple
interpretable models for few-shot learning by exploiting the
internal representations of CNNs. We adapt the VCs from
[Wang et al., 2015] to the few-shot learning setting where
the VCs are extracted from a small set of images of novel
object categories using features from CNNs trained on other
object categories. We extend the use of VCs by VC-Encoding
and observe two properties, namely category sensitivity and
spatial pattern, which leads us to propose two novel methods for few-shot learning that are simple and interpretable.
Our methods show comparable performances and much superior flexibility to the current state-of-the-art methods – it
can be applied to a range of different few-shot scenarios with
minimal re-training. In summary, we show that VCs and VCEncodings enable ordinary CNNs to perform few-shot learning tasks. Future work may include improving the quality
of the extracted VCs and extending our approach to few-shot
detection.

References
[Banerjee et al., 2005] Arindam Banerjee,
Inderjit S
Dhillon, Joydeep Ghosh, and Suvrit Sra. Clustering
on the unit hypersphere using von mises-fisher distributions.
Journal of Machine Learning Research,
6(Sep):1345–1382, 2005.
[Bau et al., 2017] David Bau, Bolei Zhou, Aditya Khosla,
Aude Oliva, and Antonio Torralba. Network dissection:
Quantifying interpretability of deep visual representations.
In Computer Vision and Pattern Recognition, 2017.
[Deng et al., 2009] Jia Deng, Wei Dong, Richard Socher, LiJia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale
hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on,
pages 248–255. IEEE, 2009.
[Finn et al., 2017] Chelsea Finn, Pieter Abbeel, and Sergey
Levine. Model-agnostic meta-learning for fast adaptation
of deep networks. arXiv preprint arXiv:1703.03400, 2017.
[Goodfellow et al., 2015] Ian J Goodfellow, Jonathon
Shlens, and Christian Szegedy. Explaining and harnessing
adversarial examples.
International Conference on
Learning Representations, 2015.
[He et al., 2015] Kaiming He, Xiangyu Zhang, Shaoqing
Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.
In Proceedings of the IEEE international conference on
computer vision, pages 1026–1034, 2015.
[Koch et al., 2015] Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot
image recognition. In ICML Deep Learning Workshop,
volume 2, 2015.
[Lake et al., 2015] Brenden M Lake, Ruslan Salakhutdinov,
and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science,
350(6266):1332–1338, 2015.
[LeCun et al., 1998] Yann LeCun, Léon Bottou, Yoshua
Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE,
86(11):2278–2324, 1998.
[Mahendran and Vedaldi, 2015] Aravindh Mahendran and
Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
5188–5196, 2015.
[Qiao et al., 2017] Siyuan Qiao, Chenxi Liu, Wei Shen,
and Alan Yuille. Few-shot image recognition by predicting parameters from activations.
arXiv preprint
arXiv:1706.03466, 2017.
[Ravi and Larochelle, 2017] Sachin Ravi and Hugo
Larochelle.
Optimization as a model for few-shot
learning. International Conference on Learning Representations, 2017.
[Sharif Razavian et al., 2014] Ali Sharif Razavian, Hossein
Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn

features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer
vision and pattern recognition workshops, pages 806–813,
2014.
[Simonyan and Zisserman, 2015] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for
large-scale image recognition. International Conference
on Learning Representations, 2015.
[Simonyan et al., 2013] Karen Simonyan, Andrea Vedaldi,
and Andrew Zisserman.
Deep inside convolutional
networks: Visualising image classification models and
saliency maps. arXiv preprint arXiv:1312.6034, 2013.
[Szegedy et al., 2017] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4,
inception-resnet and the impact of residual connections on
learning. In AAAI, pages 4278–4284, 2017.
[Vinyals et al., 2016] Oriol Vinyals, Charles Blundell, Tim
Lillicrap, Daan Wierstra, et al. Matching networks for one
shot learning. In Advances in Neural Information Processing Systems, pages 3630–3638, 2016.
[Wang et al., 2015] Jianyu Wang, Zhishuai Zhang, Cihang
Xie, Vittal Premachandran, and Alan Yuille. Unsupervised learning of object semantic parts from internal
states of cnns by population encoding. arXiv preprint
arXiv:1511.06855, 2015.
[Wang et al., 2017] Jianyu Wang, Cihang Xie, Zhishuai
Zhang, Jun Zhu, Lingxi Xie, and Alan Yuille. Detecting
semantic parts on partially occluded objects. In Proceedings of the British Machine Vision Conference (BMVC),
2017.
[Xiang et al., 2014] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object
detection in the wild. In Applications of Computer Vision
(WACV), 2014 IEEE Winter Conference on, pages 75–82.
IEEE, 2014.
[Yosinski et al., 2014] Jason Yosinski, Jeff Clune, Yoshua
Bengio, and Hod Lipson. How transferable are features
in deep neural networks? In Advances in neural information processing systems, pages 3320–3328, 2014.
[Zeiler and Fergus, 2014] Matthew D Zeiler and Rob Fergus.
Visualizing and understanding convolutional networks. In
European conference on computer vision, pages 818–833.
Springer, 2014.
[Zhou et al., 2015] Bolei Zhou, Aditya Khosla, Agata
Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns. International Conference on Learning Representations, 2015.
[Zhu et al., 2017] Zhuotun Zhu, Lingxi Xie, and Alan L.
Yuille. Object recognition with and without objects. In
Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne,
Australia, August 19-25, 2017, pages 3609–3615, 2017.

