Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN
Jiyang Gao1 Zijian (James) Guo2 Zhen Li2 Ram Nevatia1
1
2
University of Southern California
Google Research

arXiv:1711.07607v2 [cs.CV] 23 Nov 2017

jiyangga@usc.edu, {guozj, zhenli}@google.com,

Abstract

nevatia@usc.edu

Accuracy
DenseNet
ResNet

Fine-grained image labels are desirable for many computer vision applications, such as visual search or mobile AI assistant. These applications rely on image classification models that can produce hundreds of thousands
(e.g. 100K) of diversified fine-grained image labels on input images. However, training a network at this vocabulary scale is challenging, and suffers from intolerable large
model size and slow training speed, which leads to unsatisfying classification performance. A straightforward solution would be training separate expert networks (specialists), with each specialist focusing on learning one specific
vertical (e.g. cars, birds...). However, deploying dozens of
expert networks in a practical system would significantly
increase system complexity and inference latency, and consumes large amounts of computational resources. To address these challenges, we propose a Knowledge Concentration method, which effectively transfers the knowledge
from dozens of specialists (multiple teacher networks) into
one single model (one student network) to classify 100K
object categories. There are three salient aspects in our
method: (1) a multi-teacher single-student knowledge distillation framework; (2) a self-paced learning mechanism
to allow the student to learn from different teachers at various paces; (3) structurally connected layers to expand the
student network capacity with limited extra parameters. We
validate our method on OpenImage and a newly collected
dataset, Entity-Foto-Tree (EFT), with 100K categories, and
show that the proposed model performs significantly better
than the baseline generalist model.

?

Inception-V2
VGG

(a)

Alexnet
PascalVOC

ImageNet

JFT-300M

EFT

20
Accuracy

1k

20k

100k

?

# class

Specialists
(b)

Generalist
100M

2000M

# params

Figure 1. (a) Recent CNN architectures are designed and tested
on 1000-class ImageNet dataset, what will happen when we apply
them on much larger classification problems, e.g. 100K classes?
(b) Single generalist model (single model handles all classes) for
100K classes generates poor performance due to limited model
capacity; multiple specialist models (one export for a subset of
classes, e.g. 20 subsets) consume large amount of computational
resources and increase system complexity. Is there a method that
use reasonable resource and also provide good performance?

labels are far from sufficient. After all, how often do the
users need their phones to tell them that a coffee mug is
a coffee mug? Users need the visual search system to tell
them the name of a flower when they see a beautiful flower
in a park, or tell them where to buy the items when they
see fashionable items on street. To make computer vision
systems practically useful in these applications, a wide varieties of fine-grained and informative labels are required.
This paper tackles the challenge of training CNN with a
vocabulary on the scale of O(100K), 100 times larger than
the 1K labels in ImageNet [5] and 5 times larger than the
18K labels in the largest known dataset, JFT-300M [20],
as shown in Figure 1. The challenges mainly lie in two aspects: limited CNN model capacity and slow training speed.

1. Introduction
Convolutional Neural Network (CNN) models has been
rapidly improved in recent years, from AlexNet [12], VGG
[19], Inception [21], ResNet [8], to DenseNet [11]. When
applied in 1000 category classification tasks in ImageNet
[5] competition, these models have approached or exceeded
human performance. However, in many computer vision
applications, such as a mobile AI assistant, 1000 category
1

The size (number of parameters) of CNN models is limited by GPU memory. On one hand, differentiating larger
number of output classes requires stronger image representations and thus a larger bottleneck layer. On the other hand,
large number of output classes limits the size of bottleneck
layer because of limited GPU memory. For example, with a
4096-d bottleneck layer and 100K output classes, there are
more than 4096 × 100K > 400M parameters in this fully
connected layer. Together with parameters in other layers
in the model, activations, and variable replicas required by
optimizer, the training job would not fit in GPU memory.
As for training speed, training models on the JFT-300M
[20] dataset for 1 epoch needed 2 weeks on 50 Nvidia K80
GPUs and the full training schedule takes around 10 epochs
[20]. Directly training model on 100K categories, with 5
times larger vocabulary, would consume prohibitively large
amount of computational resources and much longer training time.
One straightforward solution to solve the problems is to
divide the vocabulary into semantically related subgroups
(or verticals), e.g., grouping all fine-grained birds into one
vertical while grouping all fine-grained cars into another
vertical. We then train dozens of specialist networks, with
each specialist network learns to classify limited number
of categories in one vertical. Multiple specialist networks
can be trained in parallel, which saves wall time of training.
However, deploying dozens of separate fine-grained models in a practical system would increase system complexity,
inference latency, and consume large amount of computational resources, as shown in Figure 1.
To address these challenges, we propose a multi-teacher
single-student knowledge concentration method, which effectively merges the knowledge from dozens of teachers
(i.e. specialists) into a single student model. To better transfer the knowledge from multiple teachers to a student, we
design a self-paced learning mechanism to allow the student to learn from different teachers at various paces. The
intuition of our work come from daily experience: (1) a student absorbs knowledge from multiple teachers at school on
different course subjects, and (2) a good student spends different efforts on different course subjects, since they have
different amounts of material to learn. To expand the student network capacity but limiting the of parameters, we
explore and design structurally connected layers, which includes generic units for vertical agnostic representation and
individual units for intra-vertical representations. We perform experiments on a newly collected dataset, Entity-FotoTree (EFT), with 100K categories and on OpenImage. The
EFT dataset are organized in a hierarchy and clustered into
20 verticals. Experimental results show that the proposed
knowledge concentration method outperforms the baseline
models by a large margin on both datasets.
Our contributions are three-fold:

(1) We design a novel multi-teacher single-student
knowledge distillation method to transfer knowledge from
the specialists to the generalist, and a self-paced learning
mechanism allowing the student to learn at different paces
from different teachers.
(2) We design and explore different types of structurally
connected layers to expand network capacity with limited
number of parameters.
(3) We evaluate the proposed methods on EFT and OpenImage datasets, and show significant performance improvements.

2. Related Work
Knowledge distillation. The concept of knowledge distillation is originally proposed by Hinton et al. [10], which
uses the soft targets generated by a bigger and deeper network to train a smaller and shallower network and achieves
similar performance as the deeper network. Ba et al. [3] also
demonstrated that shallow feed-forward nets can learn the
complex functions previously learned by deep nets. Romero
et al. [16] extended Hinton’s work[10] by using not only the
outputs but also the intermediate representations learned by
the teacher as hints to train the student network. The aforementioned methods fall into single-teacher single-student
manner, where the task of the teacher and the student is
the same. Rusu et al. [17] proposed a multi-teacher singlestudent policy distillation method that can distill multiple
policies of reinforcement learning agents to a single network. Our Knowledge Concentration method is also a
multi-teacher single-student framework.
Transfer learning. Fine-tuning is a common strategy
in transfer learning with neural networks [18, 24, 2, 15].
Oquab et al. [15] showed that CNN image representations
learned from large-scale annotated datasets can be efficiently transferred to other tasks with limited amount of
training data by fine-tuning the network. Li et al. [13] proposed a Learning without Forgetting method, which uses
only new task data to train the network while preserving
the original capabilities. Wang et al. [22] demonstrate that
“growing” a CNN with additional units for fine-tuning on
a new task, newly-added units should be appropriately normalized to allow for a pace of learning that is consistent
with existing units. Our self-paced learning mechanism is
related to [22], we propose a vertical level pace adjustment
mechanism and a dynamic pace initialization method, while
[22] is on node level with a fixed initialization.
Branched CNN. There is some previous work on
branched (i.e. tree-structured) CNN, which is related to
our structural connected layers. Ahmed et al. [1] presented a tree-structured network which contains multiple
branches. The branches share a common base network and
each branch classifies a subset of similar categories. Yan
et al. [23] introduced hierarchical deep CNNs by embed-

ding deep CNNs into a category hierarchy, which separates
easy classes using a coarse category classifier while distinguishing difficult classes using fine category classifiers. The
branches in tree-structured CNN can be viewed as a special
type of structurally connected layers.
Fine-grained classification. Fine-grained image classification requires the model to discern subtle differences
among similar categories. Many methods are designed
based on CNN models [25, 14, 18] that learn feature representations from data directly for classification. Cui et al.
[4] proposed a generic iterative framework for fine-grained
categorization and dataset bootstrapping which uses deep
metric learning with humans in the loop. Zhou et al. [26]
exploited the rich relationships through bipartite-graph labels. Fu et al. [7] designed a recurrent attention CNN which
recursively learns discriminative region features to discern
subtle differences for similar classes. He et al. [9] proposed
a two-stream model that combines the vision and language
for learning latent semantic representations for fine-grained
classification.

3. Knowledge Concentration
In this section, we introduce Knowledge Concentration method in three parts: (1) multi-teacher single-student
knowledge distillation framework; (2) structurally connected layers and (3) self-paced learning mechanism.

3.1. Multi-teacher Single-student Knowledge Distillation

Nb X
Nc
1 X
(log(σ(xij ))zij +log(1−σ(xij ))(1−zij ))
Nb i=1 j=1

(1)
where Nb is the batch size, Nc is the number of classes,
zij and xij are the label and output logit of sample i for class

Nb X
Nc
1 X
∗
∗
(log(σ(xij ))zij
+log(1−σ(xij ))(1−zij
))
Nb i=1 j=1

(2)
∗
where zij
and xij are the soft target and output logit of sample i for class j respectively. Note that, label smearing is not
used for student network training.
Teacher 1
Top K
Training sample x

Teacher 2
(bird)

Soft Targets

.
.
.

Teacher N

Label
Loss

Lt = −

Ls = −

Teacher Selection

The original idea of knowledge distillation proposed by
Hinton et al. [10] focuses on distilling knowledge from a
single teacher to a single student; we present a framework
for integrating knowledge from multiple teachers to a single
student.
Teacher model training. In our framework, teachers are
a set of trained specialist models. Each specialist model is a
Convolutional Neural Network (CNN) trained on a vertical
Vi . The classes in one vertical are semantically related and
usually visually confusing, e.g fine-grained cars are in one
vertical, fine-grained birds are in another vertical. We use
T (Vi ) to represent the specialist model for vertical Vi . In a
large scale dataset, such as JFT-300M [20], categories are
usually organized in a semantically hierarchical structure,
and thus not all classes are mutually exclusive. Therefore,
we use sigmoid cross-entropy loss instead of softmax crossentropy to train the teacher models.

j respectively, σ(.) is the sigmoid function. If the dataset
organizes the categories in a hierarchical structure, then the
labels are dynamically propagated to ancestor nodes, e.g. a
training sample of “golden retriever” would also be a positive sample for “dog”, “mammal” and “animal”, as “blackbird” is a sub-type of “bird”. We call this label propagation
process as “label smearing”.
Knowledge distillation from multiple teachers. Given
a training sample s and the groundtruth label z, instead of
using it for training directly, we first generate a probability
distribution p by feeding s into the corresponding teacher
(i.e. specialist) model, as shown in Figure 2. The teacher
model is selected by a vertical mapping function Fmap ,
which maps a label z to a vertical Vi . The generated probability distribution is p = G(T (Fmap (z)), s), where G is
the mapping from the image space to the label space of a
teacher model. For each sample, we keep the top K labels
and use their probabilities as soft targets z ∗ for distillation.
For the categories that are not in the top K categories of
the vertical or that are not in the selected vertical Vi , we set
their labels as 0 (i.e. negative). We use sigmoid cross entropy loss to train the student network, which is the same
for teacher network.

Student
Prediction

Figure 2. Multi-teacher single-student knowledge distillation
framework: training samples are input to the corresponding specialist model, the output probability distributions are used as soft
targets to optimize the student network.

3.2. Structurally Connected Layers
With large number of output classes, the number of parameters in the last Fully Connected (FC) layer can be prohibitively large. Therefore, shrinking the size of this bot-

tleneck layer is essential if using an FC layer structure.
In order to expand the model capacity efficiently without
blowing GPU memory, we design Structurally Connected
(SC) layers. Unlike densely connected FC layers, SC layers sparsely connect the consecutive layers and thus significantly reduce the number of parameters. We consider the
CNN architecture as “base network + 2 top layers”. The
base network contains the convolutional layers and the top
layers can be FC or SC layers. We denote the first top layer
(i.e. the layer connecting to the base network) as top-1 layer
and second top layer as top-2 layer (i.e. the layer to top of
top-1 layer and connecting the logit outputs).
Figure 3 (a) shows the architecture employing only the
fully connected layers; it is named as “FC(s1 )-FC(s2 )”,
where s1 is size for top-1 layer and s2 is the size for top2 layer. The total number of parameters in the top layers
is s2 × N + s2 × s1 + s1 × sb , where N is the number
of output classes, and sb is the output layer size of the base
network. Figure 3 (b) (c) (d) shows three types of network
architectures employing different combinations of FC and
SC layers.
Figure 3 (b) shows the network architecture where verticals have separate top-2 layers and shared top-1 layer. The
total number of parameters in the top layers is s2 × N +
M × s2 × s1s + s1s × sb , where M is the number of verticals and s1s is the size of shared top-1 layer. We name this
type of architecture “FC(s1s )-SC(s2 )”. Figure 3 (c) shows
the second type of network architecture, “SC(s1i )-SC(s2 )”,
where verticals have not only separate top-2 layers but also
separate top-1 layers. All the top-1 layers share the same
size s1i . The total number of parameters in the top layers
is s2 × N + M × s2 × s1i + M × s1i × sb . Figure 3 (d)
shows the third type, where we split the top-2 layers into a
generic part and individual parts; the generic part is shared
in all top-2 layers. As we restrict the size of the top-2 layer,
the size of individual part is decreased to s2 − x, where x
is the size of the generic part. This type is called “FC(s1s )SC(s2 , x)”. The total number of parameters in the top layers
is s2 × N + M × (s2 − sg ) × s1s + x × s1s + s1s × sb .
Actually, “FC(s1s )-SC(s2 )” can be viewed as a special case
for “FC(s1s )-SC(s2 , sg )”, where sg (generic part) is zero.
For fair comparison, we set the top-2 layer of (a), (b),
(c) and (d) to have the same size, s2 , since the number of
parameters connecting the logit layer and the top-2 layer
dominate the total number of parameters.

3.3. Self-paced Learning for Different Verticals
Different verticals have different number of classes and
different amounts of training data. Therefore, when transferring knowledge from multiple specialists into the student
network, it is desirable to control learning from different
verticals at different paces. We design a mechanism that
allows each vertical to learn at its own pace. As shown in

Figure 4, the logits are first L2-normalized inside each vertical and then multiplied by scaling factors. The scaling
factors γ can be applied either on the vertical level or on
the class level. Vertical level scaling factor is assigned to
each vertical, the final output logits is y~v = γv ||x~v ||, where
||x|| means L2-normalization, x~v is the original logits of
vertical v, γv is a single scaling factor applied on v, and y~v
is the output vector. Class level scaling factor is assigned
to each output node (i.e. class), the final output logits is
y~v = γ~v ||x~v ||, where γv is the scaling factor vector for vertical v, each class inside this vertical has a individual scaling
factor, y~v is the element-wise multiplication between γ~v and
||x~v ||.
The scaling factors γ are trainable variables. We consider the gradient deduction for a node (i.e. class) i after
L2-normalization. We use x~ˆi as the logit after normalization for node i.
x̂i = ||~x||i
(3)
After multiplying the scaling factor γi , the logit yi is
yi = γi x̂i

(4)

For class-level scaling factor, γi is different for different
classes; for vertical-level scaling factor, γi is the same for
the classes in one vertical. In back-propagation, the gradient
for xi is
∂l ∂ x̂i
1
x2
∂l
∂l
=
= γi
(
− i 3)
∂xi
∂ x̂i ∂xi
∂yi ||x||2
||x||2

(5)

We can see that larger γi will produce larger logits and also
larger gradients, thus γi adjusts the learning pace for class
i.
L2 normalization would make the logit x̂i much smaller
than before, and thus the network is hard to train. As derived
above, the ratio between ∂ x̂i and ∂xi is
x2j
∂ xˆj
1
= qP (1 − P 2 )
∂xj
xj
x2
j

1
≈ qP

x2j

≈√

1
1
q
∝√
2
N
Nv E(xj )

(6)

where Nv is the number of classes in vertical v, which contains class i. As all parameters are initialized randomly,
x2
Pj 2
xj

is close to 0. E(x2j ) is the expectation of x2j . We can
see that the ratio between
is ∂ x̂i and ∂xi is proportional to
√
√1 . Thus, we use
N
as
the initialization value for γi to
v
Nv
recover the gradient scale.

4. Evaluation
In this section, we introduce the evaluation metrics and
experimental setup, and discuss the experiments results.

#param=𝑠& ∗ 𝑁
#dim= 𝑠&
#param= 𝑠"# ∗ 𝑠&
#dim= 𝑠"#

#param=𝑠& ∗ 𝑁
#dim= 𝑠& ∗ 𝑀

#param= 𝑠& ∗ 𝑁

#param= 𝑀 ∗ 𝑠"# ∗ 𝑠&

#param= 𝑀 ∗ 𝑠"( ∗ 𝑠&

#param=𝑠& ∗ 𝑁
#dim=(𝑠&−𝑠+ + 𝑠+ ) ∗ 𝑀
#param= 𝑀 ∗ 𝑠"# ∗ (𝑠& − 𝑠+ )
+𝑠"# ∗ 𝑥
#dim=𝑠"#

#dim= 𝑠& ∗ 𝑀

#dim= 𝑠"#

#dim=𝑀 ∗ 𝑠"(

base
network

base
network

base
network

base
network

(a)

(b)

(c)

(d)

Figure 3. A regular Fully Connected (FC) layer structure is shown in (a); three types of Structurally Connected (SC) layer structures are
shown in (b),(c),(d), which are “FC(s1s )-SC(s2 )”, “SC(s1i )-SC(s2 )” and “FC(s1s )-SC(s2 , sg )” respectively. The layer dimension and
number of parameters are shown.

𝒚𝒗
𝛾"

𝒚𝒗
𝜸𝒗

L2 Norm

L2 Norm

𝒙𝒗

𝒙𝒗

vertical level

node level

Figure 4. Self-paced learning mechanism by logits L2normalization and multiplication with trainable scaling factors on
either vertical-level or node-level.

4.1. Experiments on EFT
Dataset. The Entity-Foto-Tree (EFT) dataset contains
400 million images on 100 thousand classes. The class labels are physical entities organized in a tree-like hierarchy,
which contains 20 diversified verticals: aircraft, bird, car,
dish, drink, gadget, hardware, home and garden, house,
human settlement, infrastructure, ingredient, invertebrate,
landmark, mammal, sports, watercraft, weapon, wildlife,
plant. The EFT test set contains 5 million images, with 50
held out images for each of the 100K classes.
Base network.
The base network is a modified
Inception-V2 [21]: instead of mean-pooling the “mixed5c”
layer to 1024-dimensional, we first mean-pool ”mixed5c”
with a 3 × 3 kernel to a 3 × 3 × 1024 feature map and then
reshape it to 9216 dimensional feature. All models tested
on EFT use the same base network.
Baselines. The first baseline method is a generalist CNN
that is directly trained to classify all 100K classes; For the
baseline generalist CNN, we don’t use the proposed training
method on it, but just use the standard training procedure.
The architecture shown in Figure 3 (a) is adopted, the sizes
of top-1 and top-2 layers are 4096 and 512 respectively. We
name it “FC4096-FC512”. This method serves as a performance lower bound. The second baseline method is to
train separate specialist models for each of the 20 verticals.

For the specialist models, both the sizes of top-1 and top2 layers are set to 4096 to maximize the model capacity,
and we name this architecture as ‘FC4096-FC4096”. These
specialist models serve as the performance upper bound.
Experiment setup. We set K (the number of selected
top categories for training student in knowledge distillation)
to 100. We train the models in Tensorflow with 50 GPU
workers (Nvidia P100) and 25 parameter servers. Adagrad
[6] is used to optimized the model, learning rate is set to
0.001, Batch size is set to 64. Each model is trained for
around 5 epochs, which takes approximately 40 days.
Evaluation metric. To fairly compare the performance
of the proposed methods with the baseline generalist and
the specialist models, we compare the performance on each
vertical. We use (mean) per-vertical average precision as
evaluation metric. Specifically, given a vertical, for a generalist model, we first prune the output labels by the label list
of the vertical and then calculate the mean average precision
on this vertical, which is called per-vertical average precision (pvap). Mean per-vertical average precision (mpvap) is
the average of all per-vertical average precision.
Evaluation on Multi-teacher Single-student Distillation. We compare the generalist models trained without
multi-teacher distillation and with multi-teacher distillation.
Two network architectures are tested: (1) FC4096-FC512,
which is used in baseline generalist, shown in Figure 3 (a);
(2) SC(512)-SC(512), the second type of SC layers introduced in Section 3.2, we set s1s = 512, s2 = 512. The
results are shown in Table 1. We can see that for both architectures, using multi-teacher distillation improves the performance. Specifically, multi-teacher distillation improves
the performance from 27.7 to 31.5 for FC4096-FC512 and
from 27.6 to 31.9 for SC512-SC512.
Comparison on different SC layers. We evaluate three
types of SC layer architectures shown in Figure 3 (b)(c)(d).
We fix top-2 layer size to be 512 for fair comparison.
Specifically, we first compare FC4096-FC512, FC4096SC512 and SC512-SC512 to see the depth impact of SC
layers. All three models are trained with multi-teacher dis-

Table 1. Comparison of multi-teacher single-student knowledge
distillation on EFT dataset, mean per-vertical average precision
(%) is reported.

method
FC4096-FC512
SC512-SC512

w/o distillation
27.7
27.6

w/ distillation
31.5
31.9

tillation. As shown in Table 2, FC4096-SC512 achieves the
best performance. Specifically, comparing FC4096-FC512
and FC4096-SC512, results show that FC4096-SC512 expand the model capacity efficiently and improves the performance from 31.5 to 32.9. The FC4096-SC512 structure expands the model capacity by allowing each vertical to have
its own bottleneck layer and increases the total bottleneck
layer size to 512 × N . At the same time, it removes the
cross-vertical connections and successfully avoids increasing the number of the parameters in the classification layer.
Here, removing the cross-vertical connections does not hurt
model performance because the top-2 layer is semantically
strong and the verticals are semantically separated.
Comparing FC4096-SC512 and SC512-SC512, we can
see that although SC512-SC512 contains more number of
parameters, the performance of it is lower than FC4096SC512, which shows that it is important to share the lower
level features (top-1 layer) among verticals.
Table 2. Comparison of different SC layer architectures on EFT
dataset for classification performance (mpvap %) and model size
(number of parameters).

method
FC4096-FC512
FC4096-SC512
SC512-SC512

mpvap(%)
31.5
32.9
31.9

#param
98M
135M
155M

We also test the third type of SC layer structure shown
in Figure 3 (d) by varying the size of generic part x. We
set x = 0, 128, 256, 384, 512, and the results are shown in
Table 3. It can be seen that generic part on top-2 layer is not
helpful for classification performance. The reason is that the
generic part reduces total top-2 layer size and makes top-2
layer the bottleneck of information flow.
Table 3. Comparison of generic feature size of 0, 128, 256, 384 and
512 on EFT dataset, mean per-vertical average precision (mpvap
%) is reported.

method
FC4096-SC(512,0)
FC4096-SC(512,128)
FC4096-SC(512,256)
FC4096-SC(512,384)
FC4096-SC(512,512)

mpvap(%)
32.9
32.8
32.4
31.4
31.5

#param
135M
127M
117M
108M
98M

Evaluation of self-paced learning. We first discuss
the impact of initialization value of γ, and then compare

the performance of vertical-level scaling factors and classlevel scaling factors. The architecture we use is FC4096SC512, as it achieves the best performance among all architectures. Multi-teacher distillation is adopted for all experiments. As shown in Figure 5, we compare 5 settings
of self-paced learning:
√ (1) fixed vertical-level scaling factors initialized by Nv ; (2) trainable vertical-level scaling
factors initialized by [mean = 10, dev = 1e − 3]; (3) trainable class-level scaling factors initialized by [mean = 10,
dev = 1e − 3]; (4) trainable
vertical-level scaling factors
√
initialized by [mean = Nv , dev = 1e − 3]; (5) trainable
√
class-level scaling factors initialized by [mean = Nv ,
dev = 1e − 3]. The loss optimization figures for all variants
discussed above are shown in Figure 5.
Comparing
√ (1) and (4), although (1) and (4) are both initialized as Nv and applied on vertical level, fixed scaling
factors couldn’t optimize the model well, and scaling factors should be adjustable by different verticals. Comparing (2) and (3), (4) and (5), results show that vertical-level
scaling factors generally work better than class-level scaling factors. The reason we believe is that class-level scaling
factors introduce too many additional parameters, which in
turn makes the optimization harder. Comparing (2) √
and (4),
it can be seen that initializing the scaling factors Nv is
critical for model optimization. The reason, as we discussed
before,
is that normalization changes the scale of logits and
√
Nv provides a good estimation for the logit scale.
Overall, the optimization of (1) (2) and (3) are not converged, (4) and (5) are converged. Thus, we further compare
the classification performance between (4) and (5), which
is shown in Table 5. We can see that class-level self-paced
learning have almost no effect on the classification performance. Vertical-level self-paced learning increase the performance by a large margin, which shows the effectiveness
for the proposed vertical-level self-paced learning. We think
the reason is that class-level self-paced learning introduces
too many additional parameters, which in turn make the optimization more difficult.
Comparison on verticals. In Table 4.1, we list the pervertical performance for 5 methods:(1) FC4096-FC512, this
is the baseline generalist method, (2) FC4096-FC512 with
multi-teacher distillation (FC4096-FC512+D), (3) FC4096SC512 with multi-teacher distillation (FC4096-SC512+D),
(4) FC4096-SC512 with multi-teacher distillation and vertical level self-paced learning (FC4096-SC512+D+S), (5)
20 specialist models, each model focuses on one vertical,
the model architecture is FC4096-FC4096, distllation and
self-paced learning are not used. We can see that with
the help of multi-teacher single-student distillation, structurally connected layers and self-paced learning, the performance is improved consistently for all verticals. Comparing
the baseline generalist model and the FC4096-SC512+D+S,
the performance is improved by 37% on mpvap with only

Table 4. Comparison of different methods on per-vertical average precision on EFT. In the method column, “+D” means using multiteacher distillation, “+S” means using self-paced learning. The specialist architecture is FC4096-FC4096, totally 20 of them. Abbreviation is used to fit the width of the paper: “acraft”=aircraft, “hdwr”=hardware, “hm-gd”=home&gardon, “hm-st”=human settlement, “infra”=infrastructure, “igrdt”=ingredient, “invbt”=invertebrate, “ldmk”=landmark, “mamal”=mammal, “wcraft”=watercraft, “wdlf”=wildlife
Method
FC4096-FC512
FC4096-FC512+D
FC4096-SC512+D
FC4096-SC512+D+S
Specialists

acraft
22.7
25.8
27.5
37.3
67.9

bird
33.4
42.8
44.7
51.6
82.9

car
51.8
55.5
55.9
59.2
74.2

dish
21.5
24.3
26.9
33.0
71.9

drink
30.8
34.0
35.2
38.8
72.9

gadget
38.4
42.2
44.7
52.4
82.4

hdwr
28.6
31.9
32.9
34.5
59.6

hm-gd
34.6
38.4
39.3
42.8
76.6

house
26.6
27.9
28.6
33.5
78.6

hm-st
13.6
15.8
16.7
19.0
45.9

infra
17.5
18.6
19.2
22.0
60.5

igrdt
26.8
30.3
32.1
35.9
66.4

invbt
25.9
31.5
33.3
40.5
71.5

ldmk
17.2
20.1
22.8
33.2
79.1

mamal
29.1
35.5
36.4
38.8
60.9

sports
39.2
41.9
42.5
44.1
66.0

wcraft
23.6
25.4
26.1
31.1
57.7

weapon
25.9
29.3
30.3
33.8
51.4

wdlf
15.6
24.7
26.0
33.2
59.3

plant
31.0
34.6
36.4
42.8
75.1

mean
27.7
31.5
32.9
37.9
68.0

#param
98M
98M
135M
135M
2260M

4.2. Experiments on OpenImage

Figure 5. Loss comparison of five different system variants for
self-paced learning mechanism. The lower figure is enlarged view
of the upper figure, the lower figure shows the loss curve for “train√
able vertical-level scaling factors initialized by [mean = Nv ,
dev = 1e − 3]” and
√ “trainable class-level scaling factors initialized by [mean = Nv , dev = 1e − 3]”.

Table 5. Comparison of self-paced learning on EFT dataset, mean
per-vertical average precision (mpvap %) is reported.

method
w/o self-paced learning
w/ class-level
w/ vertical-level

mpvap(%)
32.9
32.8
37.9

37% more parameters. Although 20 specialist models can
achieve 109% improvement, but the cost is 2260% more
parameters and 20 separate models, which consumes many
more computational resources.

Openimage is a much smaller dataset compared with
EFT dataset, and a normal deep CNN model [21] will not
meet a model capacity problem. Therefore, we use a smaller
and shallower network to simulate the gap between model
capacity and scale of classes, such as the gap between Inception and EFT. The general CNN architecture includes a
base network (smaller) and two top layers.
Dataset. Open Images is a dataset of about 9 million
images which have image-level labels and bounding boxes.
The validation set contains another 170 thousand images.
The total number of classes is around 6000 and the classes
are organized in a hierarchical structure. We split all classes
to 5 verticals: food, organism, man made objects, mode of
transport and others.
Base network. The network should be smaller enough
to show the model capacity restriction, so that the specialist model would perform better than generalist model on
a certain vertical. Based on our experiments, we decided
to use the “Mixed 1”+“Mixed 2”+“Mixed 3” of InceptionV2 as the base network. The “Mixed 3c” feature map
(28 × 28 × 256) is average-pooled to a 256-dimensional
feature vector, which is used as the output of the base network.
Baselines. The baseline generalist model directly classifies all 6000 categories, whose architecture is the base network plus FC512-FC512 top layers. The generalist model
serve as performance lower bound model. For specialists,
we train five separate models, each model focuses on one
vertical of categories, and the model architecture is the same
as the generalist model. The specialists serve as performance upper bound model.
Evaluation setup. We train the models using Tensorflow
by 50 GPU workers (P100) and 25 parameter severs. The
batch size is set to be 64. Adagrad is used to optimized the
networks, learning rate is set to 0.001. We use the same
evaluation metric as in EFT, which is (mean) per-vertical
average precision.
Comparison on Openimage. We compare three model
variants with the baseline: FC512-FC512 with multiteacher distillation (FC512-FC512+D), FC512-SC512
with multi-teacher distillation (FC512-SC512+D), FC512-

Table 6. Comparison of different methods on per-vertical average precision on Openimage. We report (mean) per-vertical average precision
(%). In the method column, “+D” means using multi-teacher distillation, “+S” means using self-paced learning.

Method
FC512-FC512
FC512-FC512+D
FC512-SC512+D
FC512-SC512+D+S
5*FC512-FC512(Specialists)

Ours: London Eye, Landmark, South Bank

Ours: Tank, Weapon, Tiger II

food
68.8
71.3
72.4
75.0
80.8

organism
67.5
69.0
69.5
71.9
75.6

man made objects
50.5
51.8
52.6
54.6
53.7

Ours: Bird, Perching bird, Yellow-winged
blackbird

Ours: Furniture, Chair, Egg chair

mode of transport
70.2
71.4
71.9
73.9
79.9

others
52.9
52.7
53.6
53.9
51.7

mean
62.0
63.2
64.0
65.8
68.4

Ours: Car, Muscle Car, Ford Mustang

Ours: Invertebrate, Spider, Giant house spider

Figure 6. Visual examples on EFT dataset. Top-3 predictions are listed and the green color represents the correct prediction.

SC512 with multi-teacher distillation and vertical-level
self-paced learning (FC512-SC512+D+S). The results are
shown in Table 6. Comparing FC512-FC512 and FC512FC512+D, we can see that multi-teacher single-student
knowledge distillation improves the performance from 62.0
to 63.2, which shows that multi-teacher distillation is able to
effectively transferring the knowledge to the student model.
Comparing FC512-SC512+D and FC512-FC512+D, the
results show that the structurally connected layers expand
the model capacity and thus improve the classification
performance from 63.2 to 64.0. FC512-SC512+D+S
improve the mean per-vertical average precision from 64.0
to 65.8 comparing with FC512-SC512+D, it can be seen
that vertical level self-paced learning makes the student
learn from different teachers more effectively.
It can be seen that the proposed multi-teacher distillation, self-paced learning and structurally connected layers
consistently improve the student performance on both EFT
and OpenImage. However, the performance gap between
the student model and specialists on OpenImage is smaller
than the gap on EFT. We think the main reason is that we

use a much larger bottleneck layer (top-2) for the specialist
(4096) than that for the generalist (512) on EFT, while on
OpenImage, they are the same (512).

5. Conclusion
We tackle the problem of image classification at the scale
of 100K classes. Training a single network at such a scale
is challenging due to the intolerable large model size and
slow training speed, and the performance is often unsatisfying. A straightforward solution would be training multiple
specialists. However, deploying dozens of expert networks
in a practical system significantly increase system complexity. We design a Knowledge Concentration method, which
effectively merges the knowledge from dozens of specialists (i.e. teachers) into a single student model. Specifically,
we design a multi-teacher knowledge distillation framework
and a self-paced learning mechanism to allow the student
to learn from different teachers at various paces, and structurally connected layers to expand model capacity. Our
method is evaluated on EFT and Open-Image datasets, and
performs significantly better than the baseline model.

Acknowledgements. This work would not have been
possible without the efforts of members in Image Understanding group in Google Research. The authors would
like to thank Lu Chen, Zhongli Ding and Zheyun Feng for
helping in collecting and cleaning EFT dataset; Tom Duerig
and Futang Peng for previous internal works on knowledge
distillation; Neil Alldrin and Victor Gomes for OpenImage dataset collection and evaluation; Danushen Gnanapragasam for the help in Flume pipeline; Ariel Gordon for the
TensorFlow training infra; Chen Sun, Xiangxin Zhu, Tom
Duerig, and Chuck Rosenberg for reviewing the manuscript
and providing valuable suggestions; Howard Zhou, Lucy
Gao, Pengchong Jin for the helpful discussions.

References
[1] K. Ahmed, M. H. Baig, and L. Torresani. Network of experts
for large-scale image categorization. In ECCV, 2016. 2
[2] H. Azizpour, A. S. Razavian, J. Sullivan, A. Maki, and
S. Carlsson. Factors of transferability for a generic convnet
representation. IEEE transactions on pattern analysis and
machine intelligence, 38(9):1790–1802, 2016. 2
[3] J. Ba and R. Caruana. Do deep nets really need to be deep?
In NIPS, 2014. 2
[4] Y. Cui, F. Zhou, Y. Lin, and S. Belongie. Fine-grained categorization and dataset bootstrapping using deep metric learning with humans in the loop. In CVPR, 2016. 3
[5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. FeiFei. Imagenet: A large-scale hierarchical image database. In
CVPR. IEEE, 2009. 1
[6] J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization.
Journal of Machine Learning Research, 12(Jul):2121–2159,
2011. 5
[7] J. Fu, H. Zheng, and T. Mei. Look closer to see better: Recurrent attention convolutional neural network for fine-grained
image recognition. In CVPR, 2017. 3
[8] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning
for image recognition. In CVPR, 2016. 1
[9] X. He and Y. Peng. Fine-grained image classification via
combining vision and language. In CVPR, 2017. 3
[10] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge
in a neural network. arXiv preprint arXiv:1503.02531, 2015.
2, 3
[11] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger.
Densely connected convolutional networks. In CVPR, July
2017. 1
[12] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
NIPS, 2012. 1
[13] Z. Li and D. Hoiem. Learning without forgetting. In ECCV,
2016. 2
[14] T.-Y. Lin, A. RoyChowdhury, and S. Maji. Bilinear cnn models for fine-grained visual recognition. In ICCV, 2015. 3
[15] M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Learning and
transferring mid-level image representations using convolutional neural networks. In CVPR, 2014. 2

[16] A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta,
and Y. Bengio. Fitnets: Hints for thin deep nets. arXiv
preprint arXiv:1412.6550, 2014. 2
[17] A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins,
J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu,
and R. Hadsell.
Policy distillation.
arXiv preprint
arXiv:1511.06295, 2015. 2
[18] A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson. Cnn features off-the-shelf: an astounding baseline for
recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806–
813, 2014. 2, 3
[19] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. In ICLR, 2014.
1
[20] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting
unreasonable effectiveness of data in deep learning era. In
ICCV, 2017. 1, 2, 3
[21] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.
Rethinking the inception architecture for computer vision. In
CVPR, 2016. 1, 5, 7
[22] Y.-X. Wang, D. Ramanan, and M. Hebert. Growing a brain:
Fine-tuning by increasing model capacity. In CVPR, 2017. 2
[23] Z. Yan, H. Zhang, R. Piramuthu, V. Jagadeesh, D. DeCoste,
W. Di, and Y. Yu. Hd-cnn: Hierarchical deep convolutional
neural networks for large scale visual recognition. In ICCV,
2015. 2
[24] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In NIPS, 2014.
2
[25] N. Zhang, J. Donahue, R. Girshick, and T. Darrell. Partbased r-cnns for fine-grained category detection. In ECCV,
2014. 3
[26] F. Zhou and Y. Lin. Fine-grained image classification by
exploring bipartite-graph labels. In CVPR, 2016. 3

