1

Tree-CNN: A Deep Convolutional Neural
Network for Lifelong Learning

arXiv:1802.05800v1 [cs.CV] 15 Feb 2018

Deboleena Roy, Priyadarshini Panda and Kaushik Roy, Fellow, IEEE
Abstract—In recent years, Convolutional Neural Networks (CNNs) have shown remarkable performance in many computer vision
tasks such as object recognition and detection. However, complex training issues, such as “catastrophic forgetting” and
hyper-parameter tuning, make incremental learning in CNNs a difficult challenge. In this paper, we propose a hierarchical deep neural
network, with CNNs at multiple levels, and a corresponding training method for lifelong learning. The network grows in a tree-like
manner to accommodate the new classes of data without losing the ability to identify the previously trained classes. The proposed
network was tested on CIFAR-10 and CIFAR-100 datasets, and compared against the method of fine tuning specific layers of a
conventional CNN. We obtained comparable accuracies and achieved 40% and 20% reduction in training effort in CIFAR-10 and
CIFAR 100 respectively. The network was able to organize the incoming classes of data into feature-driven super-classes. Our model
improves upon existing hierarchical CNN models by adding the capability of self-growth and also yields important observations on
feature selective classification.
Index Terms—Convolutional Neural Networks, Deep Learning, Incremental Learning, Lifelong Learning, Transfer Learning

F

1

I NTRODUCTION

D

EEP Convolutional Neural Networks (DCNNs) have
emerged as the leading architecture for large scale
image classification in recent years [1]. In 2012, Krizevsky,
et al. [2] won the ImageNet Large Scale Visual Recognition
Challenge (ISLVRC) by implementing a Deep-CNN. This
catapulted DCNNs into the spotlight, and since then, they
have dominated ISLVRC and have surpassed human level
performance on popular image datasets such as MNIST [3],
[4] and ImageNet [5].
Today, with increased access to large amount of labeled data (eg. ImageNet contains 1.2 million images
with 1000 categories), supervised learning has become the
leading paradigm in training DCNNs for image recognition.Traditionally, a DCNN is trained on a dataset containing large number of labeled images. The network learns
to extract relevant features and classify these images. This
trained model is then applied to other unlabeled images to
classify them into these specific classes. In such training,
all the training data is presented to the network during the
same training process. However, in real world, we hardly
have all the information at once. Instead, data is gathered
incrementally over time. And we, humans, also learn in a
similar incremental manner. We accumulate knowledge acquired in the past and use it to learn new things during our
lifetime. Inspired by the way humans learn, Lifelong Machine
Learning (LML) [6] is an emerging paradigm in machine
learning that tries to address this limitation. Lifelong Learning
[7], [8], is based on the principle that learning the nth task
should be easier than learning the (n − 1)th task. While,
intuitively, this makes sense; the actual implementation of
such a network faces several challenges that are discussed
further.

•

Deboleena Roy, Priyadarshini Panda and Kaushik Roy are with Department of Electrical and Computer Engineering, Purdue University, West
Lafayette, IN, 30332.

A DCNN embeds feature extraction and classification in
one coherent architecture within the same model. Modifying
one part of the parameter space immediately affects the
model globally. Another problem of incrementally training
a DCNN is the issue of “catastrophic forgetting” [9]. When
new data is fed into a DCNN, it results in the destruction
of existing features learned from earlier data. This means
when training on new data, all previous data must also be
shown to the network.
Interestingly, it has also been observed that initial layers
of DCNN learn to extract generic features, while deeper
layers learn to identify higher level features. It has been
demonstrated that initializing a new network with transferred features from a trained network can potentially boost
the performance of the new one [10]. “Off the shelf” CNNs
trained on large datasets are then fine-tuned for specific
applications [11]. However, these solutions do not train the
original network to learn the new classes. Rather a new
network is created from the old one.
In this work, we propose a network that grows hierarchically as new classes are introduced. The branching is based
on the similarity of features between new and old classes.
The initial nodes of the Tree-CNN divide the dataset into
coarse super-classes, and as we approach the leaves of the
network, finer classification is done. Such a model allows
us to leverage the convolution layers learned previously
to be used in the new bigger network. Our objective is to
present the advantages of such a network model in terms of
incremental learning over transferring trained convolution
layers and fine tuning a regular network.
The proposed Tree-CNN has the following salient features:
i.

The network is initially trained to classify images
into N classes. When an image belonging to a new
class is shown to the network, the network grows

2

ii.

iii.

iv.

to accommodate the new class.
The network grows by adding a new branch or a
new leaf node to the current structure. The decision
is based on how closely the new class resembles a
particular superclass and its sub-classes.
The objective is to reduce the training effort, which
is made up of 2 components: number of weights
updated, and the number of examples (both old
and new) required for training.
The updates are localized to a section of the tree,
while majority of the Tree-CNN is left untouched,
therefore reducing the cost of learning new classes.

The rest of the paper is organized as follows. The related
work on incremental learning in deep neural networks is
discussed in Section 2. Next, in Section 3 we explain in detail
our proposed network architecture and incremental learning method. In Section 4, we describe the two experiments
we conducted using CIFAR-10 and CIFAR-100 datasets. It
is followed by a detailed analysis of the performance of the
network and its comparison with basic transfer learning and
fine tuning in Section 5. Finally, in Section 6 we discuss the
merits and limitations of our network, conclude our findings
and suggest opportunities for future work.

2

R ELATED W ORK

The modern world of digitized data produces new information every second, thus fueling the need for systems that can
learn as new data arrives. Traditional deep neural networks
are static in that respect, and several new approaches to
incremental learning are currently being explored. “Oneshot learning” [12] is a Bayesian transfer learning technique,
that uses very few training samples to learn new classes.
Fast R-CNN [13], a popular framework for object detection,
also suffers from “catastrophic forgetting”. One way to
mitigate this issue is to use a frozen copy of the original
network compute and balance the loss when new classes
are introduced in the network [14]. "Learning without Forgetting" [15] is another method that uses only new task
data to train the network while preserving the original
capabilities. However, here the original network is trained
on an extensive dataset, such as ImageNet [5], and the new
task data is a much smaller dataset. We, on the other hand,
propose a network that knows only a handful of classes, and
grows and learns over time.
Our work draws inspiration from transfer learning. It
has been observed that initial layers of a CNN learn very
generic features [10]. The same work also demonstrated that
a new network, when initialized with “learned” layers from
a trained network, performs better. Switching initial layers
of deep neural networks with pre-defined Gabor filters have
been shown to reduce training time and offer a more energyefficient training process [16].
Common features shared between images of different
objects has been exploited to build hierarchical classifiers.
These features can be grouped semantically, such as in [17],
in which authors built an “Attention-tree”, a visual semantic
hierarchy to perform energy-efficient image classification.
“FALCON: Feature driven selective classification” [18] is
a classification methodology inspired by biological visual

attention mechanism in the brain. They use characteristic features such as color and texture to break down the
behemoth task of classifying large datasets into a set of
hierarchical classifiers.
Similar to the progression of complexity of convolutional
layers in a DCNN, the upper nodes of a hierarchical CNN
classify the images into coarse super-classes using basic
features, like grouping green-colored objects together, or
humans faces together. Then deeper nodes perform finer
discrimination, such as “boy” v/s “girl” , “apples” v/s
“oranges”, etc. Such hierarchical CNN models have shown
to perform at par or even better than standard DCNNs
[19]. “Discriminative Transfer learning” [20] is one of the
earliest works where classes are categorized hierarchically
to improve network performance. “HD-CNN” [19], is a
hierarchical CNN model that is built by exploiting the
common feature sharing aspect of images. However, in these
works, the dataset is fixed from the beginning, and prior
knowledge of all the classes and their properties is used to
build a hierarchical model. In this work, the Tree-CNN starts
out as a single root node and generates new hierarchies
to accommodate the new classes. Images belonging to the
older dataset are required during retraining, but by localizing the change to a small section of the whole network, we
try to achieve reduction in training effort. In [21], a similar
approach is applied, where the new classes are added to
the old classes, and divided into two super-classes, by using
an error-based model. The initial network is cloned to form
two new networks which are fine tuned over the two new
super-classes. While their motivation was a "divide-andconquer" approach for large datasets, we are interested in
developing a model that can incrementally grow with new
data. And, we sequentially add new data over multiple
learning stages. In the next Section, we lay out in detail our
design principle, network topology and the algorithm used
to grow the network.

3

T HE L IFELONG L EARNING M ODEL

3.1

Network Architecture

The network model is inspired from hierarchical classifiers.
It is built with nodes, connected as a directed acyclic graph.
Each node acts a classifier, predicting a label for the input
image. As per the label, the image is then passed on to the
next node which further classifies the image, until we reach
a leaf node, the last step of classification.
We begin by defining certain terms used frequently in
our model. A Node represents the building block of our
tree. Each node represents a group of 1 or more classes.
Each node has following parameters.
•
•

•

Node-ID: Each node is given a number that uniquely
identifies it.
Parent: This holds the Node-ID of the node previous
to this node. Each node can have only one parent.
A parent node can be inferred as coarse-classifier,
whereas the child node does finer classification. Except for the root node, all other nodes have parent
node. Each node can only have at max one parent
node.
Children: It is an array of Node-IDs belonging to the
nodes linked with each output neuron of the current

3

of the network to reach a specific leaf node. At each node,
beginning with the top node, the image is fed to the DCNN
associated with that node. The output node with the highest
classification probability is the next node the algorithm
moves to. If it is a leaf node, then the class associated with
that node is the predicted class. Else, the algorithm feeds the
image to the DCNN of that node. The pseudo-code for class
prediction is given under Algorithm 1.

IMAGE

Root Node
classifies
input image
into one of
the “superclasses”

Branch Node
Fine classifier

Fig. 1. A generic model of Tree-CNN: The root node predicts superclasses whereas lower nodes predict finer classes

•

•

node. The number of children each node has is equal
to the number of output neurons.
Net: This stores the convolutional neural network
that processes the input image. In case of a leaf node,
this is empty.
LT: Acronym for "Labels Transform", LT is a lookup
table that keeps track of the original label of a class,
and the output neuron number it is associated with.
For a leaf node, LT has just one record, the class label
of that node.

The root node is the highest node of the tree. The first
classification happens at this node. Each output neuron of
the root is associated with a node. Next in hierarchy is the
branch node. It has a parent and two or more children.
It performs classification for at minimum 2 classes/superclasses. The leaf node is the last level of the tree. Each leaf
node is uniquely associated to a class. No two leaf nodes
have the same class. Fig. 1 shows the root node and branch
nodes for a 2 stage classification network. Each output of
the second level branch node is a leaf node.

3.2

Algorithm

To start with, one has two options when implementing the
Tree-CNN. If there are only a handful of distinct classes
that don’t seem like they can be grouped in a hierarchical
fashion, then the network can be initialized as a single root
node. The output of the root node are the distinct classes.
However, if at the beginning we have data that is already
grouped in hierarchical manner as coarse and fine classes,
then a more detailed Tree-CNN can be initialized. The root
node will be trained to classify the images into the highest
level of super-classes. The branch nodes further down the
tree are trained to classify the images into finer classes.
Now that we have a trained Tree-CNN, we discuss how
it predicts classes and how it treats new classes. First, we
describe how the network predicts the class of an input
image. A recursive algorithm moves along the hierarchies

Algorithm 1 Class Prediction
Initialize: Im = Input Image, node = root node of the TreeCNN
procedure C LASS P REDICT(Im,node)
if num(node.children) = 0 then
. Checks if it is a leaf node
return class = node.LT
. for leaf node, LT = class label
else
NextNode = EvaluateNet(Im, node.net)
. EvaluateNet returns the node.ID of the child node
having the largest output
return C LASS P REDICT(Im,NextNode)
end if
end procedure
Next we describe the process of incremental/lifelong
learning. The Tree-CNN is already trained to classify, say,
N classes. Now, data belonging to M new classes have been
acquired and the network needs to learn to classify them
while trying to minimize the change in network structure,
and having a low training effort. A small sample of images
(∼ 10%) is selected from the training set of the new classes.
At the root node these images are fed to the DCNN, one
class at a time. We obtain a 3 dimensional matrix, O K×M×I ,
where
K = number of children of the root node
M = number of new classes
I = number of sample images per class
O(k, m, i) denotes the output of the k th output neuron for
the i th image belonging to the mth class where k ∈ [1, K],
K×M is the average of the outputs
m ∈ [1, M], and i ∈ [1, I]. O avg
over I images . Softmax is taken over O avg to obtain the
likelihood matrix L K×M which indicates how strongly an
image belonging to mth class on average associates to the
k th node.
I
Õ
O(k, m, i)
(1)
O avg (k, m) =
I
i=1
L(k, m) =

eOav g (k,m)
K
Í
eOav g (k,m)

(2)

k=1

Each column of L can be represented as a K × 1 vector, lm ,
where m ∈ [1, M] represents the M new classes. We arrange
lm in an ordered set, S, such that
S = [lm1, lm2, lm3 ...lmM ]

(3)

max(lm1 (k)) >= max(lm2 (k)) >= ... >= max(lmM (k))
k

k

k

(4)

4

We analyze each vector of the set S one by one to
determine how a new class will be appended to the root
node. There are 3 options:

ii.

iii.

FC

CONV 3

CONV 2

CONV 1

FC

FC

CONV 3

CONV 2

“Animal”

CONV 1

IMAGE
32x32x3 RGB

k

Combine one or more child nodes and the new
class to form a new child node: If there are more
than 1 child nodes that the new class has a strong
likelihood for, we can combine them to form a new
child node. Say, the top two likelihood values were
0.48,and 0.45, and at least one of them is a leaf node,
we can combine the two and the new class to form
a new child node which will be a branch node. We
can set an upper limit to the number of child nodes
that could be combined
Add the new class as a new child node: If the new
class doesn’t have a single likelihood value greater
than a threshold (lm (i) < threshold ∀i ∈ [1, K]), or
certain network restrictions are applied to prevent
addition of classes to child nodes, the network
expands horizontally by adding the new class as
a new child node. This node will be a leaf node.

Once this is complete for the M classes at the root node,
we move to the next level of the tree. The same process is
applied on the child nodes that now have new classes to
be added to them. For example, say, two new classes were
added to a child node. If child node is a leaf node, it is
changed into a branch node that now has 3 leaf nodes as
children. If the child node is a branch node, then we repeat
the process of calculating likelihood matrix and determining
how these two new classes will get added to its output. As
we can see, the decision on how to grow the tree is semisupervised. The algorithm itself decides how to grow the
tree, given the constraints by the user. We can limit parameters such as maximum children for a node, maximum
depth for the tree, etc. as per our system requirements.
Once the new classes are allotted locations in the tree, we
do supervised gradient descent based training of the nodes
that have been modified/added. Doing this saves us from
modifying the whole network, and only affected portions of
the network require retraining/fine-tuning.

4

“Root”

CONV 2

Add the new class to an existing child node: If
the value of max(lm ) is greater than a threshold
(which can be set as per requirement) indicating
a strong resemblance/association with a particular
child node, the new class is added to child node k
such that lm (k) = max
(lm (k 0 )), k 0 ∈ [1, K]
0

CONV 1

i.

“Vehicle”

T HE E XPERIMENTS

We conducted two experiments using the datasets, CIFAR10 and CIFAR-100. We used MatConvNet [22], an opensource toolbox for implementation of Convolutional Neural
Networks in MATLAB [23].
The two experiments have very different initial conditions. In the first experiment on CIFAR-10 we show that
given a trained Tree-CNN with one root node and two branch
nodes how it expands as new classes arrive. We demonstrate
a single incremental learning stage. In the second experiment with CIFAR-100, the network starts with one root node
that can classify only 10 classes, and we progressively teach
the network 90 new classes over time.

Fig. 2. Detailed network of Tree-CNN A (before incremental learning)

During training, we performed simple data augmentation. Training images were flipped horizontally at random
with a probability of 0.5 [24]. All images were whitened and
contrast normalized [24]. The activation used in all the networks is rectified linear activation ReLU, σ(x) = max(x, 0).
The networks are trained using stochastic gradient descent
with fixed momentum of 0.9. Dropout [25] is used between
the final fully connected layers, and between pooling layers to regularize the network. We also employed batchnormalization (BNORM) [26] at the output of every convolutional layer. Additionally, a weight decay λ = 0.001
was set to regularize each model. The final layer performs
softmax operation on the output of the nodes to generate
class probabilities.
4.1

Adding Multiple New Classes

4.1.1 Dataset
We used CIFAR-10 dataset [27] for this experiment. It has 10
mutually exclusive classes, namely, airplane, automobile, bird,
cat, deer, dog, frog, horse, ship, and truck. There are 50, 000
training images and 10, 000 test images equally distributed
between 10 classes. Each image is a 32 × 32 color image, thus
having 3 input channels for Red, Green and Blue pixels.
We first train the network on 6 classes, and in the next
stage of incremental learning, we train the network for the
4 remaining classes.
4.1.2 The Network
For ease of reference, we label this network as Tree-CNN A.
The root node is a DCNN with two output nodes. It will
classify the input image as either “Animals” or “Vehicles”.
Each child node has a DCNN that does finer classification.
The description of the layers in each of these sub-networks
is given in Tables 1 and 2. Fig. 2 and Fig. 4 a) depict the
initial model of Tree-CNN A.
For comparison, we took another network (Network
“B”) with a complexity level similar to two stage complexity
of this Tree-CNN. It has 4 convolutional blocks, each block
having 2 sets of 3 × 3 convolutional kernels. The network
is inspired from the architecture of VGG-net [28]. Detailed
model is given in Table 3.
4.1.3 Initial Training
CIFAR-10 has classes that can easily be grouped into 2
distinct groups, “Vehicles” and “Animals”. We start with
6 classes, 3 belonging to each group. Automobile, ship, and

5

TABLE 1
Root Node for Tree-CNN A (CIFAR-10)

TABLE 3
Network B

Input 32x32x3
Conv1
5x5x3x64 ReLU stride 1 + BNORM
[2 2] Max Pooling stride 2
Conv 2
3x3x64x128 ReLU stride 1 + BNORM
Dropout 0.5
3x3x128x128 ReLU stride 1 + BNORM
[2 2] Max Pooling stride 2
FC
Fully Connected 8x8x12x512 ReLU
Dropout 0.5
Fully Connected 1x1x512x128 ReLU
Dropout 0.5
Fully Connected 1x1x128x2 ReLU
Softmax Layer

Input 32x32x3
Conv1
3x3x3x64 ReLU stride 1 + BNORM
Dropout 0.5
3x3x64x64 ReLU stride 1 + BNORM
[2 2] Max Pooling stride 2
Conv 2
3x3x64x128 ReLU stride 1 + BNORM
Dropout 0.5
3x3x128x128 ReLU stride 1 + BNORM
[2 2] Max pooling stride 2
Conv3
3x3x128x256 ReLU stride 1 + BNORM
Dropout 0.5
3x3x256c256 ReLU stride 1 + BNORM
[2 2] Max pooling stride 2
Conv4
3x3x256x512 ReLU stride 1 + BNORM
Dropout 0.5
3x3x512x512 ReLU stride 1 + BNORM
[2 2] Avg Pooling stride 2
FC
Fully Connected 2x2x512x1024 ReLU
Dropout 0.5
Fully Connected 1x1x1024x1024 ReLU
Dropout 0.5
Fully Connected 1x1x1024xN
(N=Number of Classes)
Softmax Layer

TABLE 2
Branch Node for Tree-CNN A (CIFAR-10)
Input 32x32x3
Conv1
5x5x3x32 ReLU stride 1 + BNORM
[2 2] Max Pooling stride 2
Dropout 0.25
Conv2
5x5x32x64 ReLU stride 1 + BNORM
[2 2] Max Pooling stride 2
Dropout 0.25
Conv3
3x3x64x64 ReLU stride 1 + BNORM
[2 2] Avg Pooling stride 2
Dropout 0.25
FC
Fully Connected 4x4x64x128 ReLU
Dropout 0.5
Fully Connected 1x1x128xN ReLU
(N=number of children)
Softmax Layer

truck are grouped as “Vehicles”, while cat, dog, and horse
are grouped as “Animals”. The network at the root node is
trained to classify the images as “Animals” or “Vehicles”.
For this, the 30, 000 training images belonging to the 6
classes are re-labeled as “Animals” or “Vehicles”. The root
node is then trained for 300 epochs. The learning rate is kept
at 0.1 for first 200 epochs, then reduced by 10 times every 50
epochs. At the second level, each of the two branch nodes are
separately trained. “Animals” branch node is trained with
15, 000 training images from the 3 classes. This node further
classifies the image into dog, cat, and horse. It is trained for
300 epochs and the learning rate is same as used for the top
node. Similarly, the branch node labeled “Vehicles” is trained
to identify the 3 distinct vehicles, ship, truck, and automobile.
The root node achieves a testing accuracy of 98.73%, while
the branch nodes, “Animals” and “Vehicles”, achieve 86%
and 94.43% testing accuracy respectively. Overall, this network achieves a testing accuracy of 89.10%.
For comparison, we train our network “B” with the 6

image classes. The network is trained for 300 epochs. The
learning rate is kept at 0.1 for first 200 epochs, then reduced
by 10 times every 50 epochs. We achieve a testing accuracy
of 92.40%. The accuracy of tree-CNN A is within the range
of network “B”.
4.1.4 Incremental Learning
Next we want to train the networks to identify the remaining four classes, bird, frog, deer and airplane. There are 500
training images per class given to us. We select at random
50 images per class, and show these 200 images to the root
node. We obtain the L matrix, which is a 2 × 4 matrix with
each element li j ∈ (0, 1). The 1st row of the matrix indicates
the softmax likelihood of each of the 4 classes as being
classified as “Vehicles”, while the second row presents the
same information for “Animals”. Fig. 3 shows how likely a
new class is to being classified as “Animals” or “Vehicles”.
Note, that this is not the number of times out of 50, the
images belonging to the class are classified as one of the
two super-classes by the root node. Instead, we perform the
softmax operation (eq. 2 on the actual values of the output
layer, to see which output neuron gives a higher value for a
particular class and by how much.
As per our algorithm, airplane gets grouped with “Vehicles” while bird, deer, and frog get added to the “Animals”
coarse category. The decision making is done using Algorithm 2 based on the softmax likelihood output of the root
node, as obtained in Fig. 3. The before and after structure
of the Tree-CNN A is shown in Fig. 4 . Next the root node

6
a)

b)

ROOT

BRANCH
NODES

LEAF
NODES

ANIMAL

VEHICLE

ROOT

ANIMAL

VEHICLE

SHIP

CAT

SHIP

CAT

DOG

TRUCK

DOG

TRUCK

HORSE

AUTOMOBILE

HORSE

AUTOMOBILE

DEER

AEROPLANE

NEW
CLASSES

BIRD
FROG

Fig. 3. Softmax likelihood of new classes being classified as Animals or
Vehicles for CIFAR 10

Fig. 4. Graphical representation of Tree-CNN A a) before incremental
learning, b) after incremental learning

4.2
is retrained with 50, 000 training images from all the 10
classes(old and new) to classify them in to the two coarse
categories. The root node is trained for 250 epochs, with
learning rate 0.1 for first 100 epochs, then the learning rate is
reduced by 10 every 50 epochs. Next we train the two branch
nodes. For each of the two nodes, we train it with training
images from both new and old classes. Each of the nodes
are trained for 250 epochs and the learning rate variation is
kept same as for the root node.
Algorithm 2 Incremental Learning for Tree-CNN A
Inputs: L = 2 × 4 likelihood matrix, node = root node of
the Tree-CNN, NewClass = array of new class labels
procedure T REE G ROW(L,node)
for (col = 1, col ≤ 4, col + +) do
if L(1, col) > L(2, col) then
AddClass(NewClass(col), node.children(1))
else
AddClass(NewClass(col), node.children(2))
end if
end for
end procedure
For comparison, we apply fine tuning to our already
trained network “B”. We add the 4 new classes as 4 new
output nodes of the final layer. As described in Table 3, it
is made up of 4 convolutional blocks (CONV) and one fully
connected block (FC). For network B, 5 fine tuning strategies
have been used. Each method retrains/fine-tunes certain
layers of the network. As listed below, we set 5 different
depths of back-propagation when retraining with the whole
dataset, i.e. all 10 classes.
1.
2.
3.
4.
5.

Case I: FC
Case II: FC + CONV1
Case III: FC + CONV1 + CONV2
Case IV: FC + CONV1 + CONV2 + CONV3
Case V: FC + CONV1 + CONV2 + CONV3 + CONV4

Case V represents the extreme scenario where the previous
network is completely retrained to learn the new classes.
This is equivalent to training a new network with all the
classes.

Sequentially Adding Multiple Classes

4.2.1

Dataset

For this experiment, we use a larger dataset, CIFAR-100
[27]. It has 100 classes, 500 training and 100 testing images
per class. The image is a 32 × 32 color image with 3 input
channels for Red, Green and Blue pixels. We divide the 100
classes into 10 groups, each group having 10 classes. This
distribution is done at random. We first train the networks
with 10 classes, and then incrementally add 10 new classes
every stage. There are 9 learning stages to add the remaining
90 classes.
4.2.2

The Network

Initially, the Tree-CNN has a root node and 10 leaf nodes. We
label this network as Tree-CNN C. The root node has a DCNN
network, with 10 output nodes. The layers of the CNN are
described in detail in Table 4. In subsequent learning stages,
the network would extend branches. The DCNN model
used in branch nodes is given in Table 5. For comparison, we
use the same network “B" as used in previous experiment.
4.2.3

Initial training

For Tree-CNN C, the root node is trained to classify 10
classes. It is trained for 300 epochs, with learning rate of
0.1 for first 200 epochs. It is then reduced by 10 every 50
epochs. We obtain a testing accuracy of 84.90%. Network
“B” is trained for 300 epochs as well, and the learning rate
is same as for the root node. We achieve a testing accuracy
of 85%. The starting accuracy for the two networks is almost
the same.
4.2.4

Incremental Learning

We divided the remaining 90 classes into 9 groups, each
containing 10 classes. These classes were added to the
network in 9 incremental learning stages. At each stage, first
50 images belonging to each class are shown to the root
node and a likelihood matrix L is generated. The columns
of the matrix are used to form an ordered set S, as described
in equations 3 and 4. For this experiment, we applied the
following constraints to the system.
•

Maximum depth of the tree is 2

7
•

TABLE 4
Root Node of Tree-CNN C (CIFAR 100)
Input 32x32x3
Conv 1
5x5x3x64 ReLU stride 1 + BNORM
[2 2] Max Pooling stride 2
Conv 2
3x3x64x128 ReLU stride 1 + BNORM
Dropout 0.5
3x3x128x128 ReLU stride 1 + BNORM
[2 2] Max Pooling stride 2
Conv 3
3x3x128x256 ReLU stride 1 + BNORM
Dropout 0.5
3x3x256x256 ReLU stride 1 + BNORM
[2 2] Avg Pooling stride 2
FC
Fully Connected 4x4x256x1024 ReLU
Dropout 0.5
Fully Connected 1x1x1024x1024 ReLU
Dropout 0.5
Fully Connected 1x1x1024xN
(N = Number of Children)
Softmax Layer

Maximum number of output/children for a branch
node is 10

The depth of the tree is constrained for ease of conducting
the experiment. The algorithm then needs to be implemented only at the root node level. It was observed during
developing the algorithm that new classes tend to have
a higher softmax likelihood value for branch nodes with
higher number of children. To prevent the network from
being lop-sided, by having one very large branch, we limit
the number of children to 10 per branch node. Once the
placement of the new classes is determined we train the
root node and the modified branch nodes. The root node is
trained for 250 epochs, with learning rate 0.1 for first 100
epochs, then the learning rate is reduced by 10 every 50
epochs.
To compare, we use same fine tuning test cases on
network “B” as the previous experiment. We have 5 different
cases, each having a different depth of back-propagation, as
explained in Section 4.1.2.

5

R ESULTS

Now, we discuss the results obtained from the two sets of
experiments. To reiterate, the first experiment had only one
incremental learning stage. While, the second experiment
demonstrated how the network grew to accommodate new
data over time. The network grows 10 folds in classification
capacity, from 10 classes to 100 classes. We compare our proposed method against re-training final layers of a standard
DCNN. The methods were compared against the following
metrics
Í
•
Training Effort =
(total number of weights × total
nets

TABLE 5
Branch Node of Tree-CNN C (CIFAR 100)
Input 32x32x3
Conv 1
5x5x3x32 ReLU stride 1 + BNORM
[2 2] Max Pooling stride 2
Dropout 0.25
Conv 2
5x5x32x64 ReLU stride 1 + BNORM
[2 2] Max Pooling stride 2
Dropout 0.25
Conv 3
3x3x64x64 ReLU stride 1 + BNORM
Dropout 0.5
3x3x256x256 ReLU stride 1 + BNORM
[2 2] Avg Pooling stride 2
FC
Fully Connected 4x4x64x512 ReLU
Dropout 0.5
Fully Connected 1x1x512x128 ReLU
Dropout 0.5
Fully Connected 1x1x128xN
(N = Number of Children)
Softmax Layer

•

number of training samples)
Testing Accuracy

Every training sample generates a corresponding error that
is back-propagated through stochastic gradient descent to
update the network weights. Training Effort is the number
of weight updates that happen per training epoch. As batch
size and number of training epochs is kept the same, the
product of number of weights and the number of training
samples used gives us a good measure of the effort used in
training a network. For Tree-CNN the training effort of each
of the nodes (‘nets’) is summed together. For network “B”,
there is only one net in each case.
5.1

Adding multiple new classes (CIFAR-10)

In Fig. 5, we compare the test accuracy and the training
effort for the 5 cases of fine-tuning network “B” against
our Tree-CNN C for CIFAR-100. Retraining only FC layer
of network “B” requires the least training effort. However,
it gives us the lowest accuracy, 78.37% amongst all. And as
more classes are introduced, this method causes much loss
in accuracy, as shown with CIFAR-100 in Fig. 7. Our proposed model, Tree-CNN A has the second lowest normalized
training effort, ∼ 40% less than ‘B:Case V’, and ∼ 30% less
than ‘B:Case II’. At the same time, Tree-CNN A (86.25%) had
comparable accuracy to ‘B:Case II’ (85.02%) and ‘B:Case III’
(88.15%), while just being less than the ideal case ‘B:Case V’
by a margin of 3.76%.

8

Fig. 5. Training Effort and Test Accuracy comparison for Tree-CNN A
against Network B for CIFAR-10 dataset

Fig. 7. Testing Accuracy for CIFAR-100 over multiple learning stages

Node #3
baby

boy

girl

man

woman

poppy

butterfly

orchid

sunflower

tulip

Oak Tree

Pine Tree

Forest

StreetCar

Node #8

Node #13

Fig. 8. Examples of groups of classes formed when incrementally learning CIFAR-100
Fig. 6. Training Effort over different learning stages when sequentially
adding multiple classes (CIFAR-100)

5.2

Sequentially adding new classes (CIFAR-100)

We compare Tree-CNN C against the 5 different fine-tuning
cases of Network “B”. Fig. 6 shows the training effort for all
the test cases for CIFAR-100 experiment. We normalized the
training effort by dividing all the values with the highest
training effort. i.e. ‘B:Case V’. ‘B:Case I’ has the lowest
training effort, as we only fine tune the final fully connected
layer. However, it performs the worst in accuracy as shown
in Fig. 7. Tree-CNN C requires almost the same training effort
as ‘B:Case II’, and achieves better accuracy than ‘B:Case
II’ and ‘B:Case III’. It achieves accuracy within the same
range as ‘B:Case IV’, while requiring 20% less training effort.
‘B:Case IV’ and ‘B:Case V’ require almost similar training
effort, as the difference is only the extra training of the smallest CONV layer, i.e. the first layer. ‘B:Case V’ gives us the
best accuracy, however, that is because we are retraining the

entire network with all the images. There is no pre-trained
kernel sharing and it is as good as starting anew, thereby
requiring the highest training effort. The Tree-CNN achieves
an accuracy of 60.46% on the full CIFAR-100 dataset. It is
2.59% less than the accuracy achieved by training the full
network “B”, which gives us 63.05%. The overall accuracy
of the Tree-CNN and the network we compared with is
comparable to the range of reported accuracy of similar
sized networks, such as 67.38% by HD-CNN [19], 67.68%
by Hertel, et al [29]. Further improvements to accuracy can
be done by modifying the CNN architecture of the nodes,
and by adopting methods such as “all convolutional net”
[30], “exponential linear units” [31].
In all 6 cases, the training effort required at a particular
learning stage was greater than the effort required by the
previous stage. This is because we had to show images
belonging to old classes to avoid “catastrophic forgetting”.
Our method had lower slope for Training Effort v/s Learning Stage, as compared to all but one (Case I) of network

9

TABLE 6
Root Node of Tree-CNN C classifying CIFAR-100 classes into 17 child nodes (after 9 Learning Stages)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

trout
bridge
girl
caterpillar
lawn mower
possum
otter
poppy
sweet pepper
bicycle
lobster
clock
oak tree
skyscraper
squirrel
crocodile
wolf

dolphin
mountain
man
bottle

turtle
palm tree
woman
can

seal
tank
baby
bear

sea
willow tree
boy
skunk

mouse
castle

lizard
plain

elephant
train

mushroom
bus

maple tree
leopard

fox

tiger

telephone

rocket

porcupine

hamster
chimpanzee
aquarium fish
apple
chair

kangaroo
cloud
sunflower
orange
couch

whale
bowl
butterfly
pear
table

shrew
lion
cockroach
rose
house

dinosaur
ray fish
tulip

beaver
worm
orchid

flatfish
raccoon
spider

shark
bee
beetle

snail
rabbit
crab

bed

tractor

pickup truck

road

motorcyle

plate
streetcar

wardrobe
pine tree

keyboard
forest

television

cup

lamp

snake

camel

cattle

“B”. For Tree-CNN C, the normalized training effort incremented by 0.081 on average for every stage, while for
network “B”, the increment was 0.041, 0.086, 0.097, 0.0997,
and 0.1 for the 5 cases, in increasing order of number of
layers trained in each case.
An interesting thing to note was similar looking classes,
that were also semantically similar, were grouped under the
same branches. At the end of the 9 incremental learning
stages, the root node had 17 children nodes out of which 4
were leaf nodes and remaining 13 were branch nodes. The
details of the classes associated with each of these 17 nodes
is given in Table 6. These 13 branch nodes further had 3 to
10 leaf nodes. The final groups formed after training all the
classes is given in Table 6. Certain similar objects grouped
together is shown in Fig. 8. While there were some groups
that had objects sharing semantic similarity as well, there
were a few odd groups, such as Node 13 as shown in Fig. 8.
The groups formed is dependent partly on the sequence in
which the classes arrived. As we had limited the number of
leaf nodes of a branch to 10, even when a new class matched
strongly to an already full branch, the algorithm added it
to the the second best match, or created a new node. In
some cases, such as branch number 13, the street car was
grouped with the trees because most images of streetcar in
the dataset were green in color. Color and shape similarities
led to grouping of very semantically dissimilar objects.

6

layers of a large network, while maintaining comparable
accuracy. However, the Tree-CNN continues to grow in size
over time, and the implications of that while implementing
on hardware is an important future work. The Tree-CNN
grows in a manner such that images that share common
features are closer neighbors in the tree than those images
that are very different. The final leaf nodes, and the distance
between them can also be used as measure of how similar
or different any two images are. Such a method of training
and classification can be used to hierarchically classify large
datasets. The correlation between image similarity and the
location of the image classes in the Tree-CNN needs to be
analyzed empirically. Our proposed method, Tree-CNN, thus
offers a better incremental learning model that is based on
hierarchical classifiers and transfer learning. It could, also,
potentially be applied to large image search operations.
During inferencing, we can choose to parse the network
up to a certain depth to give us an idea of which “supercategory” the search image belongs to. Hence, it could offer
a quick solution to analyze incoming data.

ACKNOWLEDGMENT
The work was supported in part by Center for Braininspired Computing Enabling Autonomous Intelligence (CBRIC), a DARPA sponsored JUMP center, by Semiconductor
Research Corporation, and by Army Research Lab.

D ISCUSSION

In this work, we explore a method for incremental learning
when dealing with very large image databases. The motivation of this work stems from the idea that subsequent
addition of new image classes to a network should be easier
than retraining the whole network again with all classes. We
observed that expectedly, each incremental learning stage
required more effort than the previous, because images belonging to old classes needed to be shown to the CNNs. This
is due to the inherent problem of “catastrophic forgetting”
in deep neural networks. Our proposed method has a lower
rate of increase of training effort over consecutive learning
stages as compared to the effort need to fine-tune limited

R EFERENCES
[1]
[2]
[3]
[4]

W. Rawat and Z. Wang, “Deep Convolutional Neural Networks
for Image Classification: A Comprehensive Review,” Neural Computation, vol. 29, no. 9, pp. 2352–2449, 2017.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in
neural information processing systems, 2012, pp. 1097–1105.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based
learning applied to document recognition,” Proceedings of the IEEE,
vol. 86, no. 11, pp. 2278–2324, 1998.
L. Wan, M. Zeiler, S. Zhang, Y. L. Cun, and R. Fergus, “Regularization of neural networks using dropconnect,” in Proceedings of
the 30th international conference on machine learning (ICML-13), 2013,
pp. 1058–1066.

10

[5]

[6]
[7]
[8]
[9]
[10]
[11]

[12]
[13]
[14]
[15]
[16]

[17]

[18]

[19]

[20]
[21]

[22]
[23]
[24]
[25]

[26]
[27]
[28]
[29]

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet
large scale visual recognition challenge,” International Journal of
Computer Vision, vol. 115, no. 3, pp. 211–252, 2015.
Z. Chen and B. Liu, “Lifelong machine learning,” Synthesis Lectures
on Artificial Intelligence and Machine Learning, vol. 10, no. 3, pp. 1–
145, 2016.
S. Thrun and T. M. Mitchell, “Lifelong robot learning,” Robotics
and autonomous systems, vol. 15, no. 1-2, pp. 25–46, 1995.
S. Thrun, “Is learning the n-th thing any easier than learning the
first?” in Advances in neural information processing systems, 1996, pp.
640–646.
I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio,
“An empirical investigation of catastrophic forgetting in gradientbased neural networks,” arXiv preprint arXiv:1312.6211, 2013.
J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable
are features in deep neural networks?” in Advances in neural
information processing systems, 2014, pp. 3320–3328.
M. Oquab, L. Bottou, I. Laptev, and J. Sivic, “Learning and transferring mid-level image representations using convolutional neural
networks,” in Proceedings of the IEEE conference on computer vision
and pattern recognition, 2014, pp. 1717–1724.
L. Fei-Fei, R. Fergus, and P. Perona, “One-shot learning of object
categories,” IEEE transactions on pattern analysis and machine intelligence, vol. 28, no. 4, pp. 594–611, 2006.
R. Girshick, “Fast r-CNN,” in 2015 IEEE International Conference on
Computer Vision (ICCV). IEEE, dec 2015.
K. Shmelkov, C. Schmid, and K. Alahari, “Incremental learning
of object detectors without catastrophic forgetting,” in 2017 IEEE
International Conference on Computer Vision (ICCV). IEEE, oct 2017.
Z. Li and D. Hoiem, “Learning without forgetting,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
S. S. Sarwar, P. Panda, and K. Roy, “Gabor filter assisted energy
efficient fast learning Convolutional Neural Networks,” in 2017
IEEE/ACM International Symposium on Low Power Electronics and
Design (ISLPED). IEEE, jul 2017, pp. 1–6.
P. Panda and K. Roy, “Semantic driven hierarchical learning for
energy-efficient image classification,” in 2017 Design, Automation
& Test in Europe Conference & Exhibition (DATE). IEEE, 2017, pp.
1582–1587.
P. Panda, A. Ankit, P. Wijesinghe, and K. Roy, “Falcon: Feature
driven selective classification for energy-efficient image recognition,” IEEE Transactions on Computer-Aided Design of Integrated
Circuits and Systems, vol. 36, no. 12, 2017.
Z. Yan, H. Zhang, R. Piramuthu, V. Jagadeesh, D. DeCoste, W. Di,
and Y. Yu, “HD-CNN: Hierarchical deep convolutional neural networks for large scale visual recognition,” in 2015 IEEE International
Conference on Computer Vision (ICCV). IEEE, dec 2015.
N. Srivastava and R. R. Salakhutdinov, “Discriminative transfer
learning with tree-based priors,” in Advances in Neural Information
Processing Systems, 2013, pp. 2094–2102.
T. Xiao, J. Zhang, K. Yang, Y. Peng, and Z. Zhang, “Error-Driven
Incremental Learning in Deep Convolutional Neural Network for
Large-Scale Image Classification,” MM ’14 Proceedings of the ACM
International Conference on Multime, vol. d, pp. 177–186, 2014.
A. Vedaldi and K. Lenc, “MatConvNet,” in Proceedings of the 23rd
ACM international conference on Multimedia - MM 2015. ACM
Press, 2015.
MATLAB, version 9.2.0 (R2017a).
Natick, Massachusetts: The
MathWorks Inc., 2017.
I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and
Y. Bengio, “Maxout networks,” in International Conference on Machine Learning, 2013.
N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and
R. Salakhutdinov, “Dropout: a simple way to prevent neural
networks from overfitting.” Journal of machine learning research,
vol. 15, no. 1, pp. 1929–1958, 2014.
S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep
network training by reducing internal covariate shift,” in International Conference on Machine Learning, 2015, pp. 448–456.
A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” 2009.
K. Simonyan and A. Zisserman, “Very deep convolutional
networks for large-scale image recognition,” arXiv preprint
arXiv:1409.1556, 2014.
L. Hertel, E. Barth, T. Kaster, and T. Martinetz, “Deep convolutional neural networks as generic feature extractors,” in 2015

International Joint Conference on Neural Networks (IJCNN). IEEE,
jul 2015.
[30] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller,
“Striving for simplicity: The all convolutional net,” arXiv preprint
arXiv:1412.6806, 2014.
[31] D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate
deep network learning by exponential linear units (elus),” arXiv
preprint arXiv:1511.07289, 2015.

Deboleena Roy received the B.Tech+M.Tech
Dual Degree in Electronics and Electrical Communications Engineering from IIT Kharagpur,
Kharagpur, India in 2014. She is currently working towards the Ph.D. degree in Electrical and
Computer Engineering at Purdue University,
West Lafayette, IN, USA. Prior to that, she
was a Design Engineer at Qualcomm Bangalore Design Center, Bengaluru, India from 2014
to 2016. Her current research interests include
neuro-inspired algorithms for cognitive applications such as perception, reasoning and decision making.

Priyadarshini Panda received the B.E. degree
in Electrical and Electronics Engineering and the
M.Sc. degree in Physics from the Birla Institute of Technology and Science, Pilani, India,
in 2013. She is currently pursuing the Ph.D.
degree in Electrical and Computer engineering
with Purdue University, West Lafayette, IN, USA.
She was a Component Design Engineer with
Intel Corporation, Bengaluru, India, from 2013
to 2014. Prior to that, she did a six-month internship with Nvidia Graphics, Bengaluru. Her
current research interests include low-power neuromorphic computing:
energy-efficient realization of neural networks (spiking/non-spiking in
deep learning context) using novel architectures and algorithms.

Kaushik Roy received the B.Tech. degree in
Electronics and Electrical Communications Engineering from IIT Kharagpur, Kharagpur, India
and the Ph.D. degree from the Department of
Electrical and Computer Engineering, University
of Illinois at Urbana−Champaign, Champaign,
IL, USA, in 1990. He was with the Semiconductor Process and Design Center, Texas Instruments Incorporated, Dallas, TX, USA, where he
was involved in FPGA architecture development
and low-power circuit design. He joined the Faculty of Electrical and Computer Engineering, Purdue University, West
Lafayette, IN, USA, in 1993, where he is currently Edward G. Tiedemann
Jr. Distinguished Professor. He has authored more than 600 papers
in refereed journals and conferences, holds 15 patents, graduated 75
Ph.D. students, and has co-authored two books on Low Power CMOS
VLSI Design (Wiley and McGraw Hill). His current research interests include neuromorphic and cognitive computing, spintronics, device-circuit
codesign for nanoscale Silicon and non-Silicon technologies, low-power
electronics for portable computing and wireless communications, and
new computing models enabled by emerging technologies.

