Learning Mid-Level Features and Modeling Neuron Selectivity
for Image Classification

arXiv:1401.5535v2 [] 23 Jan 2014

Shu Kong
Zhuolin Jiang
Qiang Yang

AIMERYKONG @ GMAIL . COM
ZHUOLIN . JIANG @ HUAWEI . COM
QIANG . YANG @ HUAWEI . COM

Abstract
We now know that mid-level features can greatly
enhance the performance of image learning, but
how to automatically learn the image features efficiently and in an unsupervised manner is still an
open question. In this paper, we present a very efficient mid-level feature learning approach (MidFea), which only involves simple operations such
as k-means clustering, convolution, pooling, vector quantization and random projection. We explain why this simple method generates the desired features, and argue that there might be no
need to spend much time in learning low-level
feature extractors. Furthermore, to boost the performance, we propose to model the neuron selectivity (NS) principle by building an additional
layer over the mid-level features before feeding
the features into the classifier. We show that the
NS-layer learns category-specific neurons with
both bottom-up inference and top-down analysis, and thus supports fast inference for a query
image. We run extensive experiments on several public databases to demonstrate that our approach can achieve state-of-the-art performances
for face recognition, gender classification, age
estimation and object categorization. In particular, we demonstrate that our approach is more
than an order of magnitude faster than some recently proposed sparse coding based methods.

1. Introduction
Image classification performance relies on the quality of
image features. The low-level features include the common hand-crafted ones such as SIFT (Lowe, 2004) and
HOG (Dalal & Triggs, 2005), and the learned ones from
the building blocks in an unsupervised model, such as
The work is done when Shu Kong is jointly with HKUST and
Noah’s Ark Lab of Huawei Co. Ltd.

Convolutional Deep Belief Networks (CDBN) (Lee et al.,
2009) and Deconvolutional Networks (DN) (Zeiler et al.,
2011). We then take these features to improve the classification performance by generating mid-level features from
the low-level ones through further operations such as sparse
coding and pooling (Lazebnik et al., 2006; Jarrett et al.,
2009; Boureau et al., 2010).
To learn the mid-level features, these methods usually make
use of a hierarchical architecture (Zeiler et al., 2011), in
which each layer accumulates information from the layer
beneath to form more complex features. Despite their
similarity, they mainly differ in the design of nonlinearity, which is the most important part for good classification performance (Jarrett et al., 2009). Spatial pyramid
matching (SPM) based methods (Yang et al., 2009; Wang
et al., 2010; Boureau et al., 2010) apply sparse coding and
max-pooling for the nonlinearity. DN focuses on sparse
coding, pooling and unpooling for the nonlinearity (Zeiler
et al., 2011). CDBN uses sparse coding and quasi maxpooling (Lee et al., 2009). Predictive Sparse Decomposition (PSD) further introduces nonlinear absolute value rectification and local contrast normalization (Jarrett et al.,
2009). However, as pointed out by Coates et al. (Coates
et al., 2011), while some feature-learning approaches are
slightly better than others, it is not the difference of these
methods that leads to an accuracy gain. Moreover, complex methods can be easily outweighed by simpler ones that
carefully consider some specific factors, such as the receptive field size and density of extracted low-level features.
Therefore, instead of designing complicated approaches to
learn mid-level features, in this paper, we propose an efficient mid-level feature learning method (MidFea) which
consists of very simple operations, such as k-means, convolution, pooling, vector quantization and random projection as shown in Fig. 1. Through comparison with SIFT
and HMAX (Riesenhuber & Poggio, 1999), we explain
why our MidFea produces desirable features, and argue that
there might be no need to spend much time in learning lowlevel feature descriptors.
We also consider how to exploit the mid-level features

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification

Figure 1. Left panel: the overall flowchart of the proposed framework. Specifically, the proposed feed-forward MidFea learns midlevel features in a hierarchical architecture, then the Neuron Selectivity (NS) layer transforms the features into a high-level semantic
representation which is fed into the linear classifier. Right panel: demonstration of local descriptor assembling with 2 × 2 window (color
image). With the help of 3D max-pooling, our local descriptor captures the salient orientations within a cuboid.

more effectively to boost the learning performance. According to studies in neural science (Bienenstock et al.,
1982), neurons tend to selectively respond to visual signals from specific categories. Hence, we build an additional Neuron-Selectivity (NS) layer over the mid-level features as demonstrated in Fig. 1, so that the neurons can be
fired selectively and semantically for the signals from specific categories. By modeling this property as a structured
sparse learning problem that supports both top-down analysis and bottom-up inference, the NS-layer improves the
performance notably.
In summary, our contributions are two-fold. (1) We propose a simple and efficient method to learn mid-level features. We give the explanation why our approach generates
desirable features, and argue that there might be no need
to spend much time on learning the low-level features. (2)
To the best of our knowledge, this is the first time that neuron selectivity (NS) is modeled over the mid-level features
to boost classification performance. Our model builds an
NS-layer to support fast inference, which is an appealing
property in real-world application. We run extensive experiments to demonstrate our framework not only achieves
state-of-the-art results on several databases, but also runs
faster than related methods by more than an order of magnitude. We begin with describing our mid-level feature
learning approach in Section 2, followed by the proposed
NS-layer in Section 3. We then present the experimental
validation in Section 4 and conclude in Section 5.

2. Mid-Level Feature Learning
2.1. Background
The concept of mid-level features was first introduced
in (Boureau et al., 2010), meaning that features built over
low-level ones remain close to image-level information
without any need for high-level structured image description. Typically, the mid-level features (Yang et al., 2009;

Wang et al., 2010; Boureau et al., 2010) are learned via
sparse coding techniques over low-level hand-crafted ones,
such as SIFT (Lowe, 2004) and HOG (Dalal & Triggs,
2005). However, despite the promising performance in accuracy, extracting the low-level descriptors requires significant amounts of domain knowledge and human labor. The
computation is also time-consuming and lacks flexibility.
As a result, researchers have been searching for alternative
methods to learn the features for the system to be both efficient and effective.
Some impressive unsupervised feature learning methods
have been developed such as CDBN (Lee et al., 2009),
DN (Zeiler et al., 2011) and autoencoders (Hinton &
Salakhutdinov, 2006). Despite their differences, however,
empirical validation (Jarrett et al., 2009; Coates et al.,
2011) confirms several rules of thumb. First, it is the
nonlinearity in mid-level feature learning that leads to improved performance. Second, even complicated feature
learning methods are better than others, other factors may
help simpler algorithms outperform the complex ones, including more densely extracted local descriptors and suitable receptive field size. Third, despite the differences of
these methods, they consistently learn low-level features
that resemble Gabor filters, and even the simplest k-means
can produce those similar extractors (Coates et al., 2011).
Inspired by these studies, we present a very efficient midlevel feature learning approach, called MidFea, which consists of soft convolution, 3D max-pooling, local descriptor assembling, and mid-level feature generation as shown
in Fig. 1. Different from other mid-level feature learning methods that adopts SIFT and HOG, ours learns more
adaptive low-level and mid-level features, and performs
faster.

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification

2.2. The Proposed MidFea Algorithm
Soft Convolution. Our MidFea first runs k-means clustering in the training set to generate the low-level feature
extractors. Once the filters are derived, however, we do not
use them in complicated nonlinear functions (Lee et al.,
2009) or analytical sparse decompositions (Jarrett et al.,
2009; Zeiler et al., 2011). Instead, we convolve the image
with these filters to get the feature maps, which can be seen
as a 3rd-order tensor in left panel of Fig. 1. It is worth noting that, different from the simple convolution, ours is a soft
convolution (sConv) that adaptively generates sparse feature maps. sConv consists of several steps1 : convolution,
normalization and thresholding, which are demonstrated by
Fig. 2 (more illustration can be found in appendix).
There are several advantages in sConv. First, its convolutional behavior just equals to exhaustively dealing with all
possible patches in the image, and amounts to a densest local descriptor extraction. Second, normalization along the
third mode preserves local contrast information by counting statistic orientations, thus makes the resultant maps
more resistant to illumination changes, as shown in Fig. 2.
Third, the sparse property means trivial information or
background noises can be filtered out by thresholding. This
can be seen in Fig. 6 through comparison with dense SIFT
feature maps.
3D Max-Pooling. We adopt the 3D max-pooling operation (Zeiler et al., 2011) to obtain further robustness over
these sConv feature maps. Suppose we have 9 filters that
generate 9 feature maps, then 3D max-pooling operates in
a cuboid with size 2 × 2 × 2, meaning non-overlapping
2 × 2 neighborhood between every pair of previous feature
maps at the same location. The pooling leads to a single
value that is the maximum within the volume. In a macro
perspective, we get 36 new maps, each of which has the
half size compared with that of the previous ones. This
simple operation not only reduces the size of the feature
maps, but also further captures the most salient information
in a 3D neighborhood by eliminating trivial orientations. It
is worth noting that DN (Zeiler et al., 2011) also uses 3D
max-pooling for nonlinearity. However, DN is a top-down
analysis method which requires much time to derive the
feature maps by sparse coding. Ours is a feed-forward one
and thus performs very fast.
Local Descriptor Assembling. Low-level local descriptor
1

Concretely, sConv first convolves the image with the lowlevel filters, followed by normalization over all the feature maps
along the third mode into a comparative range; then thresholds the
maps element-wisely with their mean map and produces sparse
ones; finally normalizes the sparse maps along the third mode
again for the sake of subsequent operations. We test several
choices to threshold the maps, and find that using mean map to
threshold all maps always produce good results.

Figure 2. Demonstration of soft convolution: (a) three images (Georghiades et al., 2001a) of the same person under different illumination conditions; (b) convolutional feature maps of
each image displayed in each row with three different filters; (c)
normalized maps over (b); (d) thresholded maps over (c); and (e)
normalized maps over (d).

is now assembled over the resulted 36 sConv feature maps,
as demonstrated by the right panel of Fig. 1. In detail, by
splitting each feature map into overlapping 2 × 2 patches,
we can produce 4 times more maps2 . Hence, for the 36
feature maps, we can generate 144 new ones now. To have
a better perception of these feature maps, please refer to
the SIFT feature maps in sparse coding based SPM (ScSPM) (Yang et al., 2009). If we densely extract SIFT descriptors for patches centered at every pixel, then we generate 128 feature maps, each of which has the same size with
the image.
Mid-Level Feature Generation. We encode the descriptors by vector quantization (VQ) over a dictionary learned
before hand. Then, we use max-pooling on the VQ codes in
predefined partitions of the image3 , and concatenate pooled
codes into a large vector as the image representation. As
the concatenated vector usually has more than ten thousands dimensions, we use random projection (Vempala,
2004) for dimensionality reduction, and normalize the reduced vector to have unit length as the final mid-level feature. Note that random project is cheap to perform, as it
does not involve large matrix factorization opposed to other
dimensionality reduction methods. Moreover, even random project does not improve discrimination of the reduced
data, it produce performance-guaranteed results (Vempala,
2004).
2.3. Discussion
In contrast to the hand-crafted low-level descriptors such
as SIFT and HMAX, ours are learned adaptively within the
data domain in an unsupervised manner. Despite the main
difference, our model shares similarities with these hardwired features. For example, SIFT captures eight fixed
orientations over image patches, while ours not only can
do this, but also captures subtle and important information
2
Hereafter, we ignore the boundary effect for presentation
convenience.
3
For example, the image for object categorization is partitioned in spatial-pyramid scales of 2l × 2l (for l = 0, 1, 2) (Yang
et al., 2009). The partitions are different for different tasks, details
are presented in experiments.

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification

due to its adaptivity and flexibility in the learning process.
Moreover, our descriptor also resembles the HMAX feature, which is derived in a feed-forward pathway and incorporate convolution and max-pooling. But HMAX is built
in a two-layer architecture and has no sparsity in the feature maps, while ours produce more complicated and more
resilient features in a deeper architecture with soft convolution. Through the comparisons, we can say these methods
actually use the low-level descriptors to count the statistical
information related to local orientations. Therefore, it may
not need to incorporate low-level feature learning stage in
a pipeline for high-level tasks, let along learning them in a
supervised manner.
Additionally, 3D max-pooling is also adopted in adaptive
DN (Zeiler et al., 2011) for nonlinear transformation. However, its sparse feature maps are calculated through convolutional sparse coding, which means the maps have negative values that are hard to interpret. Therefore, adaptive
DN only considers their absolute values. On the contrary,
the proposed soft convolution provides non-negative elements for all feature maps, so our model is more interpretable w.r.t capturing statistic information for the orientations.

3. Neuron Selectivity Layer
Our mid-level features are generated in a purely unsupervised manner. For the sake of classification, we propose
to build an additional layer over these mid-level features to
boost the performance. This layer models the neuron selectivity principle in neural science (Bienenstock et al., 1982),
which means that certain neurons actively respond to the
signals from a specific category, while others stay unfired.
Therefore, we call this layer Neuron Selectivity (NS) layer,
and its output is fed into the classifier for classification. Let
xi ∈ Rp denote the mid-level feature of the ith input image, which belongs to one of C classes. We would like to
build an NS-layer with d neurons. Then, the NS principle can be mathematically modeled as a structured sparse
learning problem.
3.1. Bottom-Up Inference and Top-Down Analysis
Given a specific mid-level feature xi , these NS-layer neurons selectively respond to the feature xi , and generate a
set of activations hi ∈ Rd . We turn to an encoder function fW,b (xi ), where the filter W ∈ Rd×p and b ∈ Rd , to
derive the activations hi . In this paper, we use the logistic
function σ(·) to generate element-wise activations:
hi = fW,b (xi ) =σ(Wxi + b).

(1)

The encoder follows a bottom-up inference process, and the
produced activations are desired to have semantical struc-

tures. In other words, hi should have specific sparse patterns for different class labels. Before presenting how to
produce the structured activations, we first consider a topdown feedback analysis (decoder) from the activations with
the inspiration in neural science (Critchley, 2005) and successful applications in computer vision field (Jiang et al.,
2013; Zeiler et al., 2011). In this paper, we choose the simple linear decoder to fulfill this goal:
xi ≈ Dhi ,

(2)

where D ∈ Rp×d is the weight matrix that controls the linear decoder. Back to the idea of Neuron Selectivity which
can be reflected by some appropriate constraints ψ(D) and
φ(H), we unify the top-down analysis and bottom-up inference into one formulation as below:
min kX − DHk2F + αkH − fW,b (X)k2F ,

D,H,W,b

(3)
s.t. ψ(D), φ(H),

where X ∈ Rp×N stacks all the N training data in one
matrix, H ∈ Rd×N is the corresponding activations, and α
balances the effect of the two processes.
Note that the input mid-level features are normalized and
the encoder function is bounded in the range [0, 1]. Therefore, without losing generality, by considering the decoder
as a linear combination of bases in D to reconstruct the
mid-level features, we constrain the columns in D to have
unit Euclidean length, i.e. kDi k22 = 1, ∀i = 1, . . . , d.
The constraint on H is crucial and reflects the neuron selectivity property. Hence, we enforce a class-specific constraint on the activations. In other words, a particular set
of neurons should actively respond to signals from a specific class, while others stay unfired. This property can be
modeled as a structured sparse learning problem. Instead of
explicitly allocating the neurons to each class (Yang et al.,
2011; Kong & Wang, 2012; Jiang et al., 2013), we implicit
model this property via imposing an `2,1 norm to eliminate
Pd
(j)
the rows in Hc , i.e. kHc k2,1 = j=1 kHc k2 .
Besides, the activations from the same class should resemble each other, while those from different classes ought to
be as different as possible. To this end, for the activations
from the same class, say the cth class denoted by Hc , we
force them to be similar by minimizing kHc −H̄c k2F , where
H̄c is the mean vector matrix (by taking the mean vector of
activations Hc as its columns) of activations from the cth
class. At the same time, to differentiate the activations from
different classes, we drive the activations
independent as
Pas
C
possible at class level by minimizing c=1 kHTc H/c k2F ,
where H/c = [H1 , . . . , Hc−1 , Hc+1 , . . . , HC ]. Taking the
constraints on H as a Lagrangian multiplier, we have:
φ(H) =

C n
o
X
λkHc k2,1 +βkHc −H̄c k2F +γkHTc H/c k2F ,
c=1

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification

where parameters λ, β and γ control each penalty term.
The constraint defined by φ(H) has several meanings.
With the help of the first term, a sufficiently large γ decays
the third term to zero. It means the neurons automatically
break into C separate parts, and each part corresponds to
only one specific class. When β is large enough, the neurons will respond to stimuli from the same class in an identical behavior. This means the second term enforces the
mechanism to be a strong classifier. Instead of forcing the
penalty so rigorously, we set the three parameters in φ(H)
to proper values, and allow: (1) the intra-class variance to
be preserved to prevent overfitting in training process, and
(2) a few neurons to be shared across categories so that
the combination of fired neurons ensures both discrimination and compactness. Now, we arrive at our final objective
function with the Lagrangian multiplier φ(H):
min kX − DHk2F + αkH − fW,b (X)k2F + φ(H)

D,H,W

(4)

s.t. kDi k22 = 1, ∀i = 1, . . . , d.

Each variable in Eq. 5 can be alternatively optimized
through gradient descent method by fixing the others. For
detailed derivation and optimization, please refer to appendix.
3.2. Discussion
Mathematically, the proposed NS-layer, reflected by the
objective function in Eq. 5, can be seen as a fast inference for sparse coding (Jarrett et al., 2009). Different
from these methods, ours predicts the structured sparse
codes (Kavukcuoglu et al., 2009). Moreover, the decoder
term with the constraints can be seen as discriminative dictionary learning (Yang et al., 2011; Kong & Wang, 2012;
Jiang et al., 2013; Wang & Kong, 2014), which will improve the classification performance in an analytical manner. Because of the joint Sigmoid encoder, the sparse codes
are forced to be non-negative. This is a desirable property that models the intuition of combining parts to form
a whole, as opposed to the classic sparse coding which includes negative elements to cancel each other out (Hoyer,
2004).

4. Experiments
In this section, we run extensive experiments to evaluate the
proposed MidFea and NS-layer in feature learning and image classification performance4 . First, we study our MidFea and NS-layer on classification accuracy gains in a controlled way. Then, we highlight the efficiency of our framework according to the inference time for object categorization. Moreover, we carry out classification comparisons on
4
Code
is
available
at
https://github.com/aimerykong

Shu

Kong’s

GitHub:

four datasets for different tasks. Finally, we discuss some
important parameters in our framework.
Specifically, for classification comparisons, we first use a
subset of AR database (Martinez, 1998) for face recognition and gender classification. AR database consists of 50
male and 50 female subjects, and each subject has 14 images (resized to 66 × 48) captured in two sessions with illumination and expression changes. For face recognition, the
first 7 images in Session 1 of each person are used for training and the rest for testing; while for gender classification,
the first 25 male and the first 25 female individuals are used
for training and the rest for testing. we also test our framework on age estimation over the FG-NET database (Geng
et al., 2007) with images (resized to 60 × 60) spanning the
age from 0 to 69. Consistent with the literature, we use
leave-one-person-out setting for the evaluation. Finally, we
evaluate our framework on object categorization over Caltech101 (Fei-Fei et al., 2007) and Caltech256 (Griffin et al.,
2007).
Throughout the experiments, we use the linear SVM toolbox (Chang & Lin, 2011) as the classifier, and choose the
same settings to learn the low-level features, i.e. adaptively
learning 9 filters for soft convolution and each one is with
size 7 × 7. But the partitions for spatial pooling are different for different tasks, we demonstrate this along with the
experiments. Moreover, we use the classification accuracy
for face recognition, gender classification and object categorization, and the Mean Absolute Error (MAE) for age
estimation.
4.1. Accuracy Gains by MidFea and NS-Layer
To demonstrate the superiority of our MidFea and the NSlayer, we compare our model in a controlled way with selftaught (ST) learning method (Raina et al., 2007), which can
be seen as a three-layer network with the sparse codes as
the mid-level features. Face recognition over AR database
is used for the comparison, and tens of thousands face images (with alignment and rescale) downloaded from the internet are used for unsupervised feature learning. For ST,
we vary the number of dictionary bases from 200 to 1200
and the number5 of unlabeled face images from 0 up to
80, 000. We record in Fig. 3 the classification accuracies,
as well as that obtained by linear SVM on the raw image.
From the figure, we can see, consistent with the literature,
more neurons (bases) lead to better performance, and more
unlabeled data learns more reliable dictionary for ST. But
when sufficient unlabeled data are available to learn the dictionary with a certain amount of bases, the accuracy will
eventually saturate.
5
0 means no unlabeled data available. In this case, a random
matrix is used as the dictionary.

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification

one order of magnitude. The three methods use original
SIFT for the low-level feature descriptor, and thus need
more than 19 seconds to extract SIFT features in an image of 150 × 150 pixels. Even using fast SIFT extraction
approach (Lazebnik et al., 2006), the local descriptor generation is still one time slower than ours. Furthermore, ScSPM and LLC adopt sparse coding and locality-restricted
coding, hence more running time is required. Especially,
a sorting process is required before coding each local descriptor in LLC, so it is much slower. Considering the
main steps of our model are amenable to parallelization and
GPU-based implementation, we expect its applications in
the real world.
Figure 3. Demonstration of the proposed MidFea and the NSlayer in accuracy gains by comparison with self-taught learning
(ST), which can be seen as a three-layer network with the sparse
codes as the mid-level features. The x-axis indicates the amount
of unlabeled data used for learning bases by ST. The number in
bracket shows the dimensions of the mid-level features in ST.

However, when we add our NS-layer over the mid-level
features produced by ST, a notable gain is obtained. Moreover, when the features are generated by our MidFea (with
500 codewords for VQ and a single layer of 3 × 3 partition
for spatial pooling), much better performance is achieved.
With no surprise, once NS-layer is further built over our
MidFea (MidFea-NS), the best performance is recorded.
We can conclude that, the proposed MidFea learns more
robust features to represent the image, and our NS-layer
further boosts the final classification performance.
4.2. Inference on Object Categorization
To highlight the efficiency of our framework, we study
the inference time for a 150 × 150-pixel image on MATLAB in a PC with dual-core CPU, 2.50GHz, 32-bit OS and
2GB RAM. The main steps in our framework include softthreshold convolution, 3D pooling, local descriptor assembling, VQ, spatial pyramid (SP) pooling, random projection
and the inference with the classifier.
As our MidFea learns features in a bottom-up manner,
it costs much less time than top-down methods such as
adaptive DN. Specifically, adaptive DN needs more than
1 minute to produce all the feature maps and 2 more seconds with the VQ and kernel classifier. This is much slower
than ours by almost two orders of magnitude (see Table 1),
as it involves multiple iterations for decomposing the image into multi-layer feature maps. Additionally, we compare our model with three feed-forward methods, Kernel
SPM (KSPM) (Lazebnik et al., 2006), ScSPM (Yang et al.,
2009) and LLC (Wang et al., 2010). Table 1 summarizes
the detailed comparisons, which demonstrate our method
performs much faster than the compared ones by more than

4.3. Facial Attributes Recognition
We now evaluate our model on facial attributes recognition:
face recognition and gender classification on AR database,
and age estimation on FG-NET database. The linear SVM
on the raw image acts as the baseline (SVM-raw).
For fair comparison on AR database, we choose several state-of-the-art methods as their source codes are
online available, including SRC (Wright et al., 2009),
FDDL (Yang et al., 2011), LC-KSVD (Jiang et al., 2013)6
and LLC (Wang et al., 2010). We use the random
face (Wright et al., 2009) (300-dimension) as the input for
the first three methods to reproduce the results. Like our
framework, the output of LLC is also projected to 300dimension with random matrix before fed into the linear
SVM. For face recognition and gender classification, our
MidFea learns mid-level features with 500 codewords for
VQ and a single layer of 3 × 3 partition for spatial pooling.
Moreover, our NS-layer learns 300 and 10 neurons for the
two tasks, respectively. Detailed comparisons are listed in
Table 2, and some learned neurons w.r.t the two tasks are
displayed in Fig. 47 .
From Table 2, we can see with the proposed MidFea and
NS-layer, the performance outperforms the compared ones.
Furthermore, as shown in Fig. 4, even the two tasks share
the same database, the learned neurons through the NSlayer capture specific characteristics according to the task.
This intuitively demonstrates the reason why the proposed
NS-layer works for classification.
Additionally, we use FG-NET database for age estimation.
6
The setup in (Jiang et al., 2013) is different from ours, as LCKSVD originally choose 20 images per person for training and 6
for testing. We run the code in our work with the same setup as
other methods.
7
To display the learned neurons, hereafter we use the PCA for
the dimensionality reduction other than random projection. Moreover, the spatial pooling is waived here, the neurons are averaged
w.r.t one image and then projected back to the input space for the
sake of demonstration.

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification
Table 1. Comparison of detailed inference time (s). VQ, SC and SP stand for vector quantization, sparse coding and spatial pyramid
respectively. For fair of comparison, the mid-level features of all the methods are reduced to 3000 dimension by random projection.

Time (ms)
soft convolution
3D max-pooling
local descriptor
VQ or SC
SP pooling
random projection
inference
total time

KSPM

ScSPM

LLC

SIFT: 19.42 (0.24)
0.46

0.75
20.87
(1.69)

29.72

47.06
0.114
0.128
0.63
50.01
67.35
(30.83) (48.17)

Ours
0.070
0.077
0.045
0.174

0.26
0.868

Table 2. Accuracies (%) of face recognition and gender classification on the AR database.

Method

Face Recognition

Gender Classification

SVM-raw

86.7

91.3

SRC (Wright et al., 2009)
LLC (Wang et al., 2010)
FDDL (Yang et al., 2011)
LC-KSVD (Jiang et al., 2013)

90.3
91.6
92.0
91.9

92.1
92.4
93.7
93.4

MidFea
MidFea-NS

93.3
94.7

96.1
98.3

Figure 4. Left panel: original images from AR dataset and the
corresponding neurons at NS-layer for face recognition. Right
panel: the neurons learned for gender classification.

Several state-of-the-art methods are compared here, including AGES (Geng et al., 2007), RUN (Yan et al., 2007),
OHRank (Chang et al., 2011), MTWGP (Zhang & Yeung, 2010), BIF (Guo et al., 2009), and the recent CASVR (Chen et al., 2013). Except for AGES, all the methods
uses the images with Active Appearance Model (Cootes
et al., 2001). We generate 500-word codebook for VQ in
MidFea, and define the partition with a single layer of 8×8pixel overlapping grids for spatial pooling. The results
listed in Table 3 demonstrate our model performs slightly
better than the best performance ever reported by the recent CA-SVR, which is sophisticatedly designed to deal
with imbalanced data problem, e.g. there are very few images of 60 years old and above. OHRank also deals with
sparse data, but performs very slow as showed in (Chen

Figure 5. Neurons learned on the FG-NET for age estimation.
The response values reveal that different ages do have an association to specific neurons, and we can see the neurons selectively
response to the facial textures in older people as they have more
wrinkles.

et al., 2013). The BIF method resembles ours as it uses
the (hand-designed) biologically-inspired feature (Riesenhuber & Poggio, 1999) for the mid-level features, which,
however, are generated in a shallower architecture. Some
neurons shown in Fig. 5 demonstrate our model reveals the
age information through the wrinkles on face.
4.4. Object Categorization
We also evaluate our model on two popular databases,
Caltech101 and Caltech256. For both databases, we randomly select 30 images of each category for training and
the rest for testing, and each image is resized to no larger

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification
Table 3. MAE of age estimation on the FG-NET database.

Method

MAE

Method

MAE

SVM-raw

7.86

AGES (Geng et al., 2007)
OHRank (Chang et al., 2011)
BIF (Guo et al., 2009)

6.77
4.85
4.77

RUN (Yan et al., 2007)
MTWGP (Zhang & Yeung, 2010)
CA-SVR (Chen et al., 2013)

5.78
4.83
4.67

MidFea

4.73

MidFea-NS

4.62

Figure 6. Visual comparison of local descriptor feature maps of our model and ScSPM (Yang et al., 2009) for three images from
Caltech101. The nine learned filters are presented in (a). (b) shows the three original images, whose feature maps generated by our
model (upper row) and ScSPM (bottom row) are presented in panel (c). Note that the last image in each row is the average of all feature
maps. From the averaged map, we can see the SIFT map distributes attention uniformly over the image, while ours mainly focuses on
the object.

than 150 × 150-pixel resolution with preserved aspect ratio. The compared methods include both recent unsupervised feature learning methods and well-known methods
with hand-crafted SIFT features. The former methods include CDBN (Lee et al., 2009), adaptive DN (Zeiler et al.,
2011), and PSD (Jarrett et al., 2009). The latter include the
KSPM (Lazebnik et al., 2006), ScSPM (Yang et al., 2009)
and LLC (Wang et al., 2010)8 . For all the methods, midlevel features are generated with a 1000-word codebook
for VQ or sparse coding, then reduced to 3000 dimension
by random projection9 . In our model, the NS-layer learns
8

LLC exploits a much larger dictionary (2048/4096 in Caltech101/256) for coding, we run the code with the same setup in
our work.
9
We try reducing the original data to 5000/4000/3000 dimension, and the performance does not drop at all than that on the
original data. But when we reduce them to 2000/1000, the accu-

2040 and 5120 neurons in total for the two database respectively, assuming an average of 20 neurons associate to
one specific category. The classic 3-layer-pyramid partition
(l = 0, 1, 2) for pooling is used. Detailed comparisons are
listed in Table 4 for Caltech101 with the test time on the
same image and Table 5 for Caltech256 with the standard
deviations over 10 trials.
It is easy to see that our method outperforms those with the
sophisticated SIFT descriptor. Most importantly, the inference speed of our model is much faster than the compared
ones by more than an order of magnitude. Actually, we can
assemble the SIFT descriptor of every possible patch in one
image as 128 feature maps in ScSPM. Therefore, we can
compare the feature maps between ScSPM and ours to inturacy drops by approximate 5% and 10%. That is why we choose
3000 in our work.

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification
Table 4. Accuracies and inference time over Caltech101. For timing comparison, the mid-level features of all methods are reduced to
3000 dimension by random projection. We observe this reduction does not effect the accuracies for these methods. The time in bracket
is achieved by improved SIFT extraction method in (Lazebnik et al., 2006).

Method

ACC (%)

Infer. Time (s)

Jarrett et al. (PSD) (Jarrett et al., 2009)
Lee et al. (CDBN) (Lee et al., 2009)
Zhang et al. (Zhang et al., 2006)
Zeiler et al. (adaptive DN) (Zeiler et al., 2011)

65.6
65.4
66.2
71.0

54.34

Lazebnik et al. (SPM) (Lazebnik et al., 2006)
Yang et al. (ScSPM) (Yang et al., 2009)
Wang et al. (LLC) (Wang et al., 2010)
Jiang et al. (LC-KSVD) (Jiang et al., 2013)

64.6
73.2
73.4
73.6

21.25 (2.08)
50.07 (30.89)
67.41 (48.23)
25.90 (6.72)

MidFea
MidFea-NS

73.8
74.7

0.69
0.70

Table 5. Accuracies of object categorization on Caltech256.

Method

ACC (%)

KSPM (Yang et al., 2009)
Yang et al. (ScSPM) (Yang et al., 2009)
Wang et al. (LLC) (Wang et al., 2010)
Zeiler et al. (adaptive DN) (Zeiler et al., 2011)
Boiman et al. (NN) (Boiman et al., 2008)
Jiang et al. (LC-KSVD) (Jiang et al., 2013)

29.5 ± 0.5
34.0 ± 0.6
30.4
33.2 ± 0.8
38.0
34.3
Figure 7. (a) The choices of α, β, γ and λ vs. accuracy over AR

MidFea
MidFea-NS

36.6 ± 0.5 database for face recognition; (b) The number of neurons in NSlayer vs. accuracy over AR database for gender classification.
38.8 ± 0.4

itively see the superiority of our model. Fig. 6 (a) displays
the learned low-level feature extractors, (c) shows some
feature maps (full feature maps presented in appendix) of
three images in (b). Furthermore, we average all the feature maps and show the averaged one in the last column of
Fig. 6 (c). It is easy to see SIFT feature maps in ScSPM incorporates more cluttered background, while ours focuses
more on the object and discards the noisy region to a large
extent. We attribute this to the proposed soft convolution
step.
Admittedly, better performances are reported in literature
on the datasets. For example, 75.7% accuracy is achieved
on Caltech101 in (Boureau et al., 2010) with SIFT feature,
sparse coding and intersection kernel; Sohn et al. (Sohn
et al., 2011) obtain 77.8% on Caltech101 with much larger
codebook (4096 atoms), more complicated sparse coding
technique and the SIFT feature; Varma et al. (Varma &
Ray, 2007) achieve state-of-the-art on Caltech256 as reported in (Boiman et al., 2008) with multiple hand-crafted
feature types and sophisticated techniques w.r.t the dataset.
However, our model achieves impressive performance by
simply learning the features in a purely unsupervised man-

ner from low-level to mid-level. It is far from the optimal
mechanism for the task of classification. Therefore, based
on our framework, more sophisticated task-driven methods
can be combined to address specific vision-based problems.
4.5. Parameter Discussion
Now we discuss the crucial parameters in our model
(Eq. 5), including α, β, γ and λ. Moreover, the number of
neurons in the NS-layer is also studied. Fig. 7 (a) presents
the curve of accuracy vs. each parameter on AR database
for face classification (with same setting of face recognition
experiment). It is easy to see the classification accuracy is
not sensitive to these parameters, and remains stable in a
large range of them. As well, these hyper-parameters reveal that the terms in the objective function indeed brings
performance gains. Furthermore, we show the accuracies
vs. the neuron number in NS-layer on gender classification
in Fig. 7 (b), as the data is sufficient for this task (with same
setting of gender classification experiment). We can see the
accuracy peaks with a small number of neurons, say 6. This
demonstrates the effectiveness of NS-layer that serves for
classification at a high level.

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification

5. Conclusion with Future Work
In this paper, we present a simple and efficient method for
learning mid-level features. With comparison of the sophisticated hand-crafted features, we explain why the proposed
method produces the desired features. We argue that there
might be no need to spend much time in learning the lowlevel feature extractors. Furthermore, we propose to build
an additional layer at higher level for classification. The
layer models the principle of neuron selectivity in neural
science. Given an image, our MidFea produces the midlevel features very quickly in a feed-forward process, and
the NS-layer also supports fast bottom-up inference. As a
result, our model performs faster than others by more than
an order of magnitude, and achieves comparable or even
state-of-the-art classification performance as demonstrated
by experiments.
Despite the effectiveness of the proposed feature learning
method, the performance gains in the public databases,
especially the Caltech256 for object categorization, still
remains marginal. The reason is obvious that the foreground objects are of large changes of appearance, translation and scale. Therefore, with more sophisticated features
like (Krizhevsky et al., 2012; Donahue et al., 2014) and our
paper, an explicit mechanism to deal with these changes is
still solicited (Kong & Jiang, 2014).

References

Cootes, Timothy F., Edwards, Gareth J., and Taylor,
Christopher J. Active appearance models. PAMI, 23(6):
681–685, 2001.
Critchley, Hugo D. Neural mechanisms of autonomic, affective, and cognitive integration. Journal of Comparative Neurology, 493(1):154–166, 2005.
Dalal, Navneet and Triggs, Bill. Histograms of oriented
gradients for human detection. In CVPR, 2005.
Donahue, Jeff, Jia, Yangqing, Vinyals, Oriol, Hoffman,
Judy, Zhang, Ning, Tzeng, Eric, and Darrell, Trevor. Decaf: A deep convolutional activation feature for generic
visual recognition. In ICML, 2014.
Fei-Fei, Li, Fergus, Rob, and Perona, Pietro. Learning generative visual models from few training examples: An
incremental bayesian approach tested on 101 object categories. CVIU, 106(1):59–70, 2007.
Geng, Xin, Zhou, Zhi-Hua, and Smith-Miles, Kate. Automatic age estimation based on facial aging patterns.
PAMI, 2007.
Georghiades, Athinodoros S., Belhumeur, Peter N., and
Kriegman, David J. From few to many: Illumination
cone models for face recognition under variable lighting
and pose. PAMI, 2001a.

Bienenstock, Elie L, Cooper, Leon N, and Munro, Paul W.
Theory for the development of neuron selectivity: orientation specificity and binocular interaction in visual cortex. The Journal of Neuroscience, 2(1):32–48, 1982.

Georghiades, Athinodoros S., Belhumeur, Peter N., and
Kriegman, David J. From few to many: Illumination
cone models for face recognition under variable lighting
and pose. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 23(6):643–660, 2001b.

Boiman, Oren, Shechtman, Eli, and Irani, Michal. In defense of nearest-neighbor based image classification. In
CVPR, 2008.

Griffin, Gregory, Holub, Alex, and Perona, Pietro. Caltech256 object category dataset. 2007.

Boureau, Y-L, Bach, Francis, LeCun, Yann, and Ponce,
Jean. Learning mid-level features for recognition. In
CVPR, 2010.

Guo, Guodong, Mu, Guowang, Fu, Yun, and Huang,
Thomas S. Human age estimation using bio-inspired features. In CVPR, 2009.

Chang, Chih-Chung and Lin, Chih-Jen. Libsvm: a library
for support vector machines. TIST, 2011.

Hinton, Geoffrey E and Salakhutdinov, Ruslan R. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507, 2006.

Chang, Kuang Yu, Chen, Chu Song, and Hung, Yi Ping.
Ordinal hyperplanes ranker with cost sensitivities for age
estimation. In CVPR, 2011.

Hoyer, Patrik O. Non-negative matrix factorization with
sparseness constraints. JMLR, 5:1457–1469, 2004.

Chen, Ke, Gong, Shaogang, Xiang, Tao, Mary, Queen, and
Loy, Chen Change. Cumulative attribute space for age
and crowd density estimation. In CVPR, 2013.

Jarrett, Kevin, Kavukcuoglu, Koray, Ranzato, MarcAurelio, and LeCun, Yann. What is the best multi-stage architecture for object recognition? In ICCV, 2009.

Coates, Adam, Ng, Andrew Y, and Lee, Honglak. An
analysis of single-layer networks in unsupervised feature
learning. In AISTATS, 2011.

Jiang, Zhuolin, Lin, Zhe, and Davis, L. Label consistent
k-svd: Learning a discriminative dictionary for recognition. PAMI, 2013.

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification

Kavukcuoglu, Koray, Ranzato, Marc’Aurelio, Fergus, Rob,
and LeCun, Yann. Learning invariant features through
topographic filter maps. In CVPR, 2009.

Wright, John, Yang, Allen Y, Ganesh, Arvind, Sastry,
Shankar S, and Ma, Yi. Robust face recognition via
sparse representation. PAMI, 2009.

Kong, Shu and Jiang, Zhuolin. Collaborative Receptive
Field Learning. arXiv, 2014.

Yan, Shuicheng, Wang, Huan, Tang, Xiaoou, and Huang,
Thomas S. Learning auto-structured regressor from uncertain nonnegative labels. In ICCV, 2007.

Kong, Shu and Wang, Donghui. A dictionary learning approach for classification: separating the particularity and
the commonality. In ECCV, 2012.
Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E.
Imagenet classification with deep convolutional neural
networks. In NIPS, 2012.
Lazebnik, Svetlana, Schmid, Cordelia, and Ponce, Jean.
Beyond bags of features: Spatial pyramid matching for
recognizing natural scene categories. In CVPR, 2006.
Lee, Honglak, Grosse, Roger, Ranganath, Rajesh, and Ng,
Andrew Y. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In ICML, 2009.
Lowe, David G. Distinctive image features from scaleinvariant keypoints. IJCV, 60(2):91–110, 2004.
Martinez, Aleix M. The ar face database. CVC Technical
Report, 24, 1998.
Raina, Rajat, Battle, Alexis, Lee, Honglak, Packer, Benjamin, and Ng, Andrew Y. Self-taught learning: transfer
learning from unlabeled data. In ICML, 2007.
Riesenhuber, Maximilian and Poggio, Tomaso. Hierarchical models of object recognition in cortex. Nature neuroscience, 2(11), 1999.
Sohn, Kihyuk, Jung, Dae Yon, Lee, Honglak, and Hero,
Alfred O. Efficient learning of sparse, distributed, convolutional feature representations for object recognition.
In ICCV, 2011.
Varma, Manik and Ray, Debajyoti. Learning the discriminative power-invariance trade-off. In ICCV, 2007.
Vempala, Santosh S. The random projection method, volume 65. AMS Bookstore, 2004.
Wang, Donghui and Kong, Shu. A classification-oriented
dictionary learning model: Explicitly learning the particularity and commonality across categories. Pattern
Recognition, 47(2):885–898, 2014.
Wang, Jinjun, Yang, Jianchao, Yu, Kai, Lv, Fengjun,
Huang, Thomas, and Gong, Yihong.
Localityconstrained linear coding for image classification. In
CVPR, 2010.

Yang, Jianchao, Yu, Kai, Gong, Yihong, and Huang,
Thomas. Linear spatial pyramid matching using sparse
coding for image classification. In CVPR, 2009.
Yang, Meng, Zhang, Lei, Feng, Xiangchu, and Zhang,
David. Fisher discrimination dictionary learning for
sparse representation. In ICCV, 2011.
Zeiler, Matthew D, Taylor, Graham W, and Fergus, Rob.
Adaptive deconvolutional networks for mid and high
level feature learning. In ICCV, 2011.
Zhang, Hao, Berg, Alexander C, Maire, Michael, and Malik, Jitendra. Svm-knn: Discriminative nearest neighbor
classification for visual category recognition. In CVPR,
2006.
Zhang, Yu and Yeung, Dit-Yan. Multi-task warped gaussian process for personalized age estimation. In CVPR,
2010.

Appendix: Optimization at Neuron
Selectivity Layer
For presentational convenience, we write the proposed objective function in Neuron Selectivity (NS) layer as below:
min kX − DHk2F + αkH − fW,b (X)k2F +

D,H,W,b

C n
o
X
(5)
λkHc k2,1 + βkHc − H̄c k2F + γkHTc H/c k2F
c=1

s.t. kDi k22 = 1, ∀i = 1, . . . , d.

Each variable in the above objective function can be alternatively optimized by fixing the others through stochastic
gradient descent method (SGD).
Updating D
Specifically, we apply SGD to update D by fixing the others, and its gradient can be easily computed as below:
∇D = −2XHT + 2DHHT .

(6)

Alternatively, when bases number in D is not prohibitively
large, we can analytically update D = XHT (HHT )−1 .

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification

Then, we normalize each column of D to have unit Euclidean length. Note that this step will not pose any negative affects on the overall performance, as the newly updated H, on which D is only dependent, can adaptively
deal with this scaling change.
Updating Hc class by class
Prior to optimizing Hc , which is the responses in the NSlayer to the data from the cth class, we calculate the mean
vector and get H̄c first. Then, we update Hc as:
H∗c

= argminkXc −

DHc k2F

+ αkHc −

fW (Xc )k2F

Hc

(7)

+βkH̄c − Hc k2F + γkHT/c Hc k2F + λH kHc k2,1 ,

where√Xc stacks all
cth class. Let G̃c =
√ data from the(p+2d+N
−Nc )×Nc
(Xc );√ β H̄c ; 0] ∈ R
, and
[Xc ; αfW√
√ T
(p+2d+N −Nc )×d
, in
Q̃c = [D; αI; βI; γH/c ] ∈ R
which 0 is a zero matrix with appropriate size. We rewrite
the above function as:
g(Hc ) = kG̃c − Q̃c Hc k2F + λH kHc k2,1 .

(8)

We use SGD to optimize Hc , and the partial derivative of g
w.r.t Hc is calculated as:
∇Hc =

−2Q̃Tc G̃c

+

2Q̃Tc Q̃c Hc

+ λH CHc ,

(9)

where C is a diagonal matrix with its ith diagonal element
as:
C[i, i] =

1
(i)

kHc k2

,

(10)

(i)

where Hc is the ith row of Hc .
Therefore, Hc can be optimized between solving Eq. 10
and Eq. 9 for a couple of times.
Updating W and b
With the newly calculated H, we update W and b as:
{W∗ , b∗ } = argminkH − σ(WX + b1T )k2F ,
W,b

Σ

∇b =2 (Σ − H)

Σ


(1 − Σ) XT ,

(1 − Σ) 1,

wherein ” ” means Hadamard product.

Initialization and Algorithmic Summary
Usually, a good initialization for the variables can lead to
fast convergence. For example, the linear decoder or the
dictionary D can be pre-trained among each category. Let
D = [D1 , . . . , Dc , . . . , DC ] ∈ Rp×d , in which Dc are
the neurons that only response to data from the cth class.
Then, we can merely run k-means clustering among the
data pool of class c to obtain Dc . Then, activations Hc can
be initialized through the initialized D. Specifically, for a
datum x, we calculate a vector z ∈ Rd with its ith element
as:
similarity(di , x)
,
zi = PCk
j=1 similarity(dj , x)

(13)

in which we can simply define similarity(m, n) =
1
km−nk2 . After obtaining z, we get the initialized h =
z
kzk2 . With the initialized H, both W and H can be then
pre-trained for their initialization.
However, the above initialization method lack flexibility,
because the allocation of D to each category must be predefined by hand. Therefore, we can also simply initialized
all the variables with non-negative random matrices, which
serve the purpose of symmetry breaking. Empirically, we
observe this random initialization does not mean inferior
performance at all, but requires more time to converge. We
owe it to that, even through our framework is a deep architecture, the NS-layer only incorporates one hidden layer,
hence random initialization works quite well.
The overall steps (with gradient descent method) of the NSlayer is summarized in Algorithm 1.

(11)

where 1 is a vector/matrix with all elements equaling 1 and
appropriate size. As W and b cannot be derived directly,
we use the SGD to update them. With simple derivations,
by denoting the element-wise operation Σ = σ(WX +
b1T ), we have the gradient of W and b as:
∇W =2 (Σ − H)

Algorithm 1 Algorithmic Summary at NS layer
Require: training set, Xc for class c = 1, . . . , C
1: initialize D, H, W and b randomly
2: while stop criterion is not reached do
3:
update D with its gradient in Eq. 6
4:
update Hc with its gradient in Eq. 9
5:
update W and b with their gradient in Eq. 12
6: end whilereturn the learned W, b, D

(12)

Appendix: Highlight in Mid-Level Feature
Learning
Due to limited space of the paper, we explain more advantages for the proposed mid-level feature learning method
(MidFea), especially the soft convolution (sConv). As
stated in the paper, there are two significant advantages in
sConv:
• Normalization along the third-mode preserves local
contrast information by counting statistic orientation,

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification
selective filters

and thus makes the resultant feature maps more resistant to illumination changes (illustrated by Fig. 9 to
11).
• The sparse property means trivial information or background noises can be filtered out (illustrated by Fig. 12
to 14).
Therefore, in this section, we illustrate the above two highlights through displaying full feature maps that appear in
the paper. All figures are preferably viewed by 300%
zooming in.
Illumination Invariance via Soft Convolution
Illumination-invariant feature maps are obtained by the
proposed soft convolution (sConv), which only requires
several simple operations: convolution, normalization and
soft thresholding. Put it in a pipeline, sConv does the following:
1. convoluting the original image with several low-level
feature extractors (filters) and producing several (convolutional) feature maps, and stacks them in a thirdorder tensor;
2. normalizing the feature maps along the third mode of
the tensorial maps, scaling them into a comparative
range for the sake of the subsequent soft thresholding
operation;
3. calculating the mean map along the third mode among
the feature maps, and using it to threshold all the
maps; (This stage produces sparse feature maps.)
4. employing a further normalization operation along the
soft-threshold maps. (Similar to step 2, this stage will
benefit the subsequent local descriptor assembling operation.)
To study this illumination invariance property, we use Extended YaleB database (Georghiades et al., 2001b) for the
demonstration. This database contains 2,414 frontal face
images of 38 individuals, and is challenging due to varying illumination conditions and expressions. Therefore, we
use it to illustrate our proposed soft convolution for generating illumination-invariant feature maps. Moreover, the
low-level feature extractors are shown in Fig. 8.
We randomly select three different persons for demonstration, and show three different images with different illumination for each person as Fig. 9, Fig. 10 and Fig. 11.
In detail, in each figure, (a) to (f) are the original image,
the convolutional feature maps, normalized convolutional
maps, soft-threshold maps, normalized maps over the softthreshold ones, and 3D max-pooling feature maps, respec-

Figure 8. Low-level feature extractors (filters) for soft convolution.

tively. It is easy to see the generated feature maps effectively handle the illumination changes. In other word, the
generated low-level feature (maps) is invariant to illumination.
We also evaluate our framework over this database for face
recognition. Consistent with the settings in the literature
of face recognition on this database , half images per individual are randomly selected for training and the rest for
test. Our method leads to an average of 99.6% accuracy. To
the best of our knowledge, it is the best performance ever
reported with this setting on the database.
Noise Filtering in Sparse Feature Maps
The second highlight is that the generated feature maps
can filter out background noises to some extent. This
property is brought out by the proposed soft convolution
and 3D max-pooling. In detail, trivial or weak orientation responses in the feature maps are removed through
soft threshold operation, while strong ones are kept and
strengthened by the normalization.
To show this property, we display the full feature maps
that appear in the experimental section of the paper. Three
images in Caltech101 database (Fei-Fei et al., 2007) are
randomly selected, which come from three different categories, Faces (image0117), Motorbikes (image0672) and
panda (image0033). Here, the so-called feature maps consist of the assembled local descriptors (144 dimensions) of
each patch. Similarly, the compared SIFT feature maps in
SPM scheme consist of SIFT descriptors (128 dimensions)
of all patches centered at every pixel in the image, therefore one image generates 128 feature maps. The three sets
of feature maps of the images are display in Fig. 12, Fig. 13
and Fig. 14, respectively. In each figure, (a) shows the original gray-scale image, (b) and (c) are the averaged feature
maps of SIFT maps and ours that are demonstrated in (d)
and (e), respectively. From the figures, we can see the SIFT
map distributes attention uniformly over the image, while
ours mainly focuses on the target object by constructing
mid-level features.

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification

Figure 9. The first person. (a) original image, (b) convolutional feature maps, (c) normalization over convolutional feature maps, (d)
soft-threshold maps, (e) normalization over the soft-threshold maps, (f) 3D max-pooling.

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification

Figure 10. The second person. (a) original image, (b) convolutional feature maps, (c) normalization over convolutional feature maps,
(d) soft-threshold maps, (e) normalization over the soft-threshold maps, (f) 3D max-pooling.

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification

Figure 11. The third person. (a) original image, (b) convolutional feature maps, (c) normalization over convolutional feature maps, (d)
soft-threshold maps, (e) normalization over the soft-threshold maps, (f) 3D max-pooling.

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification

Figure 12. Faces image. (a) original image, (b) average over all maps of SIFT descriptor, (c) average over all maps of the proposed local
descriptor, (d) the full SIFT feature maps, (e) the full maps of the proposed descriptors.

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification

Figure 13. Motorbikes image. (a) original image, (b) average over all maps of SIFT descriptor, (c) average over all maps of the proposed
local descriptor, (d) the full SIFT feature maps, (e) the full maps of the proposed descriptors.

Learning Mid-Level Features and Modeling Neuron Selectivity for Image Classification

Figure 14. Panda image. (a) original image, (b) average over all maps of SIFT descriptor, (c) average over all maps of the proposed
local descriptor, (d) the full SIFT feature maps, (e) the full maps of the proposed descriptors.

