Under review as a conference paper at ICLR 2016

M EDIATED E XPERTS
N ETWORKS

FOR

D EEP C ONVOLUTIONAL

arXiv:1511.06072v1 [cs.LG] 19 Nov 2015

Sebastian Agethen, Winston H. Hsu
National Taiwan University
Graduate Institute of Networking and Multimedia

A BSTRACT
We present a new supervised architecture termed Mediated Mixture-of-Experts
(MMoE) that allows us to improve classification accuracy of Deep Convolutional
Networks (DCN). Our architecture achieves this with the help of expert networks:
A network is trained on a disjoint subset of a given dataset and then run in parallel
to other experts during deployment. A mediator is employed if experts contradict
each other. This allows our framework to naturally support incremental learning, as adding new classes requires (re-)training of the new expert only. We also
propose two measures to control computational complexity: An early-stopping
mechanism halts experts that have low confidence in their prediction. The system allows to trade-off accuracy and complexity without further retraining. We
also suggest to share low-level convolutional layers between experts in an effort
to avoid computation of a near-duplicate feature set. We evaluate our system on a
popular dataset and report improved accuracy compared to a single model of same
configuration.

1

I NTRODUCTION

Deep learning methods in general, and Deep Convolutional Neural Networks (DCN) in particular,
have seen a surge in popularity among researchers over the past decade or so. While the application
of early DCN was limited to simple tasks such as hand-written digit recognition Cun et al. (1990),
modern state-of-the-art methods can meet or exceed human-level performance on far more complex
tasks, including generic image classification He et al. (2015); Krizhevsky et al. (2012); Szegedy et al.
(2014). Two factors that have contributed to this development are the availability of large-scale image corpora like ImageNet Russakovsky et al. (2014), and the wide-spread availability of GPU-based
computing hardware which renders computationally expensive model training procedures feasible.
Experts systems are known to improve the classification accuracy of neural networks even further
and have been studied extensively. Previous work on Mixture of Experts (MoE) Jacobs et al. (1991)
is however flawed in two aspects: First, computational complexity in MoE is a multiple of that of a
traditional model. To this end, we propose early-stopping of experts. Our mechanism evaluates each
expert’s confidence and is influenced by a hyper-parameter, allowing a trade-off between complexity
and classification accuracy without the need for retraining. Second, as recently pointed out in Xiao
et al. (2014), training becomes more difficult as datasets are getting larger. Intuitively, we expect
that an expert has extensive knowledge on a limited area of expertise, i.e., a subset of the training
data, and thereby naturally avoids the issue. Previous MoE systems are trained on full datasets,
however, as difficulties arise when a decision is needed on which expert to trust. To remedy this
issue, we introduce a mediator, see Figure 1. The mediator has a full classifier, allowing it to
arbitrate conflicting predictions when multiple experts are confident.
In this paper we propose a new architecture which we name Mediated Mixture of Experts (MMoE)
that aims to increase the classification accuracy of a general DCN. We briefly discuss related research in Section 2 before formally defining our method in Section 3. We follow by summarizing
the outcome of our experimental evaluation in Section 4 and give some concluding remarks in Section 5.
1

Under review as a conference paper at ICLR 2016

Conv 1
Conv 2

Mediator
All classes

Conv 3

Conv 4

Conv 4

IC24

Conv 5

Conv 5

FC6
Classifier
Softmax
w1

Conv 4

ICN4
Expert N: |CN | classes

Expert 1: |C1 | classes

IC14

Conv 4

FC6
Classifier
Softmax
0 w2 = 0

Conv 5
FC6
Classifier
Softmax
w3

Conv 5
FC6
Classifier
Softmax
wM ed

Weighted Averaging

Figure 1: N experts are trained on their corresponding superclasses Ci . At inference time, confidence is measured in confidence modules ICij . In this example, expert two is stopped at layer Conv
4 due to low confidence and its activations are set to zero. A mediator with full classifier improves
accuracy of the system should more than one expert remain.

2

R ELATED W ORK

A great deal of literature on deep learning and in particular on DCN has been published in recent
years. Image classification with deep learning has considerably profited from the availability of
large datasets (in terms of both number of classes and images). In this paper, our choice falls on the
popular ImageNet Russakovsky et al. (2014) dataset. Both a 22K classes and a 1K classes version
exist of this dataset, we evaluate our system on the latter.
AlexNet by Krizhevsky et al. Krizhevsky et al. (2012) achieved a significant increase in classification
performance over traditional methods on ImageNet, i.e., an top-1 accuracy of 62.5%, showing the
potential of deep learning for large datasets such as ImageNet. Their success is arguably founded on
three factors: Depth, Rectified Linear Units (ReLU) as non-linearity and data augmentations. We
use CaffeNet, a variation of this model, as a baseline to evaluate our system.
A hierarchical architecture for incremental learning was presented in Xiao et al. (2014). The authors suggested to use a Branching layer to determine the superclass a specific problem belongs
to. Following this, a leaf model provided fine-grained classification. We follow their work insofar
that we also propose to use experts (leaf models) for fine-grained classification. However, we avoid
branching as it introduces a conditional error that cannot be recovered. Branching error also poses a
scalability issue, as it grows with the number of superclasses. Finally, the prediction process in their
work has large redundancy: Full models are used for both branching prediction and fine-grained
prediction.
Mixture of experts (MoE), as first proposed in Jacobs et al. (1991), have been well-known for a
while. A large deal of work has been done in this area and an overview can be found in Yuksel et al.
(2012). Generally, a number N of expert networks are trained together on a dataset. The experts
learn discriminant features and thereby improve the overall accuracy of the system. While this can
be used to improve classification results, the drawback lies in the parallel execution of all N experts,
which results in longer runtime. In contrast to our proposed method, experts are trained on the full
2

Under review as a conference paper at ICLR 2016

Table 1: Branching networks introduce additional error: We trained such a network with two experts
on superclasses C1 , C2 as in Sec. 4.1. Due to branching, the average accuracy on the full dataset
C1 ∪ C2 remains below the traditional model. However, experts on their own have encouragingly
higher accuracy on their area of expertise than a traditional model, see columns C1 , C2 . Note that
the accuracy shown here was achieved with a slim configuration of CaffeNet, i.e., the FC layers were
reduced from 4096 to 512 neurons.
C1 ∪ C2
93.2 %
–
–
49.151%
49.35 %

Name
Branching
Expert 1
Expert 2
Average
Baseline Krizhevsky et al. (2012)

Class C1
93.35%
46.262 %
–
–
41.52 %

Class C2
93.04%
–
59.202%
–
55.96 %

dataset and have a complete classifier. This is disadvantageous for incremental learning, a scenario
in which new classes are added over time, as all experts need to be retrained on the new data.
The work of Yosinski et al. (2014) discusses generality and specialization of convolutional features,
giving insight into how deep learning works. The same phenomenon has also been discussed in
Zeiler & Fergus (2013) in the context of network visualization. In particular, both works show that
the first few layers of a DCN produce general Gabor-like features, i.e., lines and blobs. It is only in
the higher layers that these features become more class-specific. Inspired by this fact, instead of N
times producing nearly identical general features, we can compute a single set of these features.

3

P ROPOSED M ETHOD

When tackling a problem, a divide-and-conquer approach can often be helpful: In deep learning, we
can train experts on small problems such that the accuracy improves. This has been used in MoE
Jacobs et al. (1991): Experts are trained competitively in the hope that they automatically learn
discriminant features. A gating network combines the prediction results.
Each expert’s area of expertise can also be designed with help of prior knowledge. We can utilize
two methods to gain this knowledge: Spectral Clustering (as done in Xiao et al. (2014)) and explicit
hierarchies (such as the ImageNet dataset provides), see also Sec. 4.1. In both cases the i-th expert
is then trained exclusively on the subset of data in that superclass, which we denote as Ci .
3.1

S IMPLE B RANCHING MODEL

We briefly discuss a branching network as in Xiao et al. (2014), which predicts the superclass of an
image. The branching network is a DCN such as AlexNet Krizhevsky et al. (2012) with the number
of outputs in the classifier reduced to N , the number of superclasses. In the branching decision, the
expert corresponding to the highest activation is selected to then obtain a fine-grained prediction.
We trained such a system with two slim1 experts, as seen in Table 1. Even though we used two
experts only, classification accuracy suffered from large branching errors: Should the wrong expert
be chosen, the error cannot be recovered. Nevertheless, we also see that single experts show superior
performance, confirming that we are on the right track. The model we discuss in the following is an
attempt at reducing branching errors.
3.2

B RANCHED EXPERTS WITH EARLY STOPPING

We propose the following framework as depicted in Fig. 1: The input is passed through a number
of convolutional layers to generate general features, as defined in Yosinski et al. (2014). These
layers are shared between all experts and could encompass the first two or three layers in the case
of CaffeNet. Higher layer features have a higher degree of specialization, and therefore need to be
finetuned for each expert. Parallel execution of these layers is costly, however, in particular for large
1
By reducing the number of neurons of both layers FC6 and FC7 to 512, in the hope to keep parameter
complexity low.

3

Under review as a conference paper at ICLR 2016

(a)

(b)

(c)

Figure 2: Mediation process: Experts (a) and (b) both show strong opinions that contradict each
other. Mediator (c) is able to solve the conflict (here in favor of expert (a)).
Table 2: Mean and standard deviation of confidence scores in cases of correct and incorrect branching decisions. Incorrect decisions very often imply a low confidence value, allowing us to decide
whether additional experts need to be used to solve the task.
Branching Mean Std.-dev.
Correct
6.1
3.44
False
1.8
1.67
N . To this end we propose a confidence module, which determines the confidence of whether the
expert is able to solve the problem. In particular, an expert i is deemed to have low confidence (in
layer j) if for its given score sji and a threshold T :
 
max sjk − sji ≥ T, 1 ≤ i, k ≤ N
(1)
k6=i

In the following, we implement the confidence module simply as a fully connected layer with N
outputs, trained on the complete dataset with the original labels replaced by a superclass label lCi ∈
[0, N − 1]. Note the placement of the confidence module: While lower layers have too general
features to compute a reliable score, early stopping in higher layers diminishes the resource saving
effect. We suggest an intermediate position such as after Conv4.
Let ujCi be the activation of the confidence module in layer j of the expert on Ci . The appropriate
score is then simply the i-th component of the activation vector:
sji = ujCi ,i
To support our decision for a simple threshold, we run a simple branching network (as described
above) on a test set, and measure the magnitude of the left-hand side of Equation (1). We then note
whether the branching decision is correct or not. As is to be expected, bad branching decisions occur
when this magnitude is small, while correct decisions often have higher confidence, see Table 2.
3.3

M EDIATOR

Given that the inequality in (1) holds, an expert is deemed to have low confidence and stopped.
His corresponding activations in the final layer are set to zero. So far, the system described above
performs slightly worse than a single model with same configuration. This is due to those instances,
where more than one expert is active: An expert trained on a different superclass could “mistake”
the input for a particular fine-grained class of his own domain, leading to conflicting opinions. In
these cases, we found it helpful to add a mediator, that is, a slim model trained on all 1000 classes
to arbitrate opinions, see Figure 2. The resulting architecture then outperforms the single model
baseline. Finally, the softmax probabilities of each expert are weighted and averaged: The slim
1000-classes mediator is only added when more than one expert is executed and weighted with
wM ed . We set wM ed = 0.6 if executed or wM ed = 0 otherwise. Experts are weighted according to
their confidence scores normalized to [0, 1] and subsequently scaled by (1 − wM ed ).
4

Under review as a conference paper at ICLR 2016

Our framework can also be used in the context of incremental learning. To add a new superclass
CN +1 , we simply train a single expert on the new data. Following this, the confidence modules of
other experts need to be fine-tuned to accommodate for the new superclass. We stress that this is
significantly faster than training a full network, as the confidence module only encompasses a single
layer with N outputs, rendering it very shallow and slim. In our experiments, 3 epochs were found
to be sufficient. In order to update the mediator network, we can simply add extra neurons to the
output and finetune the network, see the Flat Increment technique in Xiao et al. (2014).

4

E VALUATION

The evaluation is performed on the ImageNet 1K dataset. We begin by describing how the superclasses corresponding to each expert are formed.
4.1

S UPERCLASS CONSTRUCTION

Superclasses can be defined in an automated or manual fashion: In the former case Spectral Clustering, as was done in Xiao et al. (2014), is a suitable choice. Alternatively, we can traverse the
hierarchy available with ImageNet and join several leaf classes that are conceptually related. This
provides a convenient advantage in an incremental learning scenario: We simply train one or more
new experts for new data. Contrary, when using Spectral Clustering, it may be necessary to retrain
some of the existing experts as well.
Due to space limitations, we limit ourselves to manually defined superclasses and N = 2. We choose
C1 as “artifact” (with synset: n00021939) and C2 as all remaining classes, resulting in a split of 517
vs. 483 leaf classes with roughly 660K and 620K images each. In other words, this simulates adding
“artifact” data to an existing model. In the following, we refer to this as the Hierarchical set.
4.2

R ESULTS ON H IERARCHICAL SET

Network configuration. We evaluate our system on three configurations: First, a slim configuration
of CaffeNet, where both FC layers are reduced to 512 neurons. In our second configuration we drop
layer FC7 altogether, while reducing FC6 to 512 neurons. Finally, our third configuration uses
the default CaffeNet configuration with both fully connected layers containing 4096 neurons. All
configurations were finetuned from the CaffeNet model available with Caffe, see CaffeNet (2014).

Figure 3: Top-1 accuracy of our proposed system in relation to threshold T , here on the slim seven
layer configuration with the first three layers shared. The expected number of parameters to be
loaded equals that of traditional MoE systems for T = 4.
5

Under review as a conference paper at ICLR 2016

Table 3: Top-1 accuracy, and stopping probabilities p1 , p2 of both experts for T = 4 under three
different configurations.
Name
MMoE (slim, 7 layers)
MMoE (slim, 8 layers)
MMoE (default)
MMoE (slim, 7 layers, shared)
Unmediated (slim, 7 layers, shared)
Baseline (slim, 7 layers)
Baseline (slim, 8 layers)
Baseline CaffeNet (2014)

Acc.
0.5618
0.5296
0.5855
0.5361
0.5052
0.5344
0.4948
0.5584
(0.574) 2

p1
0.2213
0.1805
0.2225
0.2122
0.2122
–
–
–

p2
0.1512
0.1545
0.17144
0.1801
0.1801
–
–
–

System accuracy. MMoE improves the top-1 classification accuracy in all three configurations. The
result is strongly dependent on the value of threshold T , see Figure 3. Classification accuracy peaks
at T = 6, for which our system outperforms the baseline over 2.8% and 2.7%2 (in the slim and
default configuration respectively).
Early stopping. Experts are rarely stopped falsely, see Figure 4. For T ≥ 3, a true expert is
incorrectly stopped in less than 1% of all cases, an improvement over the branching error discussed
in Section 3. Table 3 shows the probability of how often an expert is stopped for all configurations.
Mediator impact. To show the importance of the Mediator, we train a slim configuration lacking
mediation. As such, the system only reaches a top-1 accuracy of 50.52%, about 3.1% lower than the
same configuration with mediator.
Complexity vs. Threshold. The choice of T also influences how often an expert is run. We show
this by evaluating the average number of parameters that need to be loaded, see Figure 3. For
T = 4, the expected number of parameters is close to the number of parameters when using traditional Mixture-of-Experts, that is, the complexity of N experts alone. The number of computational
operations, dominated by the Convolutional layers, can be significantly reduced by layer sharing as
discussed earlier.
Impact of sharing layers. We also tested how sharing different numbers of convolutional layers
affects performance. To achieve this, we finetuned the experts from the mediator, while keeping
the lowest k convolutional layers frozen, i.e., setting the learning rate to 0. We show our results
for all three aforementioned configurations in Figure 5. Performance drops with larger number of
shared layers, as is to be expected. The default configuration, having 4096 neurons in each fully
connected layer, shows smaller impact: Accuracy drops by two percent going from no shared to
fully shared layers, leading us to the assumption that the fully connected layers are able to mitigate
the impact. The other two slim configurations however suffer a loss of more than 5% when sharing
all convolutional layers.

5

C ONCLUSION AND F UTURE W ORK

We propose a mediated expert system for deep convolutional networks that enables experts to learn
on small partitions of a training set, a case given in an incremental learning scenario. In detail,
experts can be stopped early, where the stopping point is controlled by a single hyper-parameter,
allowing to adapt to different circumstances in terms of availability of resources. Furthermore, we
avoid the branching error that occurs in prior work when training on partitioned datasets.
In order to better underline the benefits of our proposal, two points are left for our future work. First,
a more thorough evaluation is necessary that was not possible in the scope of this paper. Second,
we believe that the mediator concept can be developed further in order to reduce computational
complexity to a higher degree.
2
Using caffe’s test command, we measured CaffeNet’s top-1 accuracy at 55.84% on the validation set,
the reported accuracy is 57.4% however.

6

Under review as a conference paper at ICLR 2016

Figure 4: Probability that the true expert is falsely
stopped in relation to threshold T . The error
is significantly lower than the branching error in
Section 3 (6.8%).

Figure 5: Performance of MMoE at T = 4 with
varying number of shared layers. The default configuration (CaffeNet) shows greater resilience as
its performance only drops by 2 percent.

R EFERENCES
CaffeNet. Bvlc caffe on github. https://github.com/BVLC/caffe/tree/master/
models/bvlc_reference_caffenet, 2014. Accessed: 2015-09-23.
Cun, Le, Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D.
Handwritten digit recognition with a back-propagation network. In Advances in Neural Information Processing Systems, pp. 396–404. Morgan Kaufmann, 1990.
He, K., Zhang, X., Ren, S., and Sun, J. Delving Deep into Rectifiers: Surpassing Human-Level
Performance on ImageNet Classification. ArXiv e-prints, February 2015.
Jacobs, Robert A., Jordan, Michael I., Nowlan, Steven J., and Hinton, Geoffrey E. Adaptive mixtures
of local experts. Neural Comput., 3(1):79–87, March 1991. ISSN 0899-7667. doi: 10.1162/neco.
1991.3.1.79. URL http://dx.doi.org/10.1162/neco.1991.3.1.79.
Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with
deep convolutional neural networks.
In Pereira, F., Burges, C.J.C., Bottou, L., and
Weinberger, K.Q. (eds.), Advances in Neural Information Processing Systems 25, pp.
1097–1105. Curran Associates, Inc., 2012. URL http://papers.nips.cc/paper/
4824-imagenet-classification-with-deep-convolutional-neural-networks.
pdf.
Russakovsky, Olga, Deng, Jia, Su, Hao, Krause, Jonathan, Satheesh, Sanjeev, Ma, Sean, Huang,
Zhiheng, Karpathy, Andrej, Khosla, Aditya, Bernstein, Michael, Berg, Alexander C., and Fei-Fei,
Li. ImageNet Large Scale Visual Recognition Challenge, 2014.
Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir,
Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions.
CoRR, abs/1409.4842, 2014. URL http://arxiv.org/abs/1409.4842.
Xiao, Tianjun, Zhang, Jiaxing, Yang, Kuiyuan, Peng, Yuxin, and Zhang, Zheng. Error-driven incremental learning in deep convolutional neural network for large-scale image classification. In
Proceedings of the ACM International Conference on Multimedia, MM ’14, pp. 177–186, New
York, NY, USA, 2014. ACM. ISBN 978-1-4503-3063-3. doi: 10.1145/2647868.2654926. URL
http://doi.acm.org/10.1145/2647868.2654926.
Yosinski, Jason, Clune, Jeff, Bengio, Yoshua, and Lipson, Hod. How transferable are features
in deep neural networks? CoRR, abs/1411.1792, 2014. URL http://arxiv.org/abs/
1411.1792.
7

Under review as a conference paper at ICLR 2016

Yuksel, S.E., Wilson, J.N., and Gader, P.D. Twenty years of mixture of experts. Neural Networks
and Learning Systems, IEEE Transactions on, 23(8):1177–1193, Aug 2012. ISSN 2162-237X.
doi: 10.1109/TNNLS.2012.2200299.
Zeiler, Matthew D. and Fergus, Rob. Visualizing and understanding convolutional networks. CoRR,
abs/1311.2901, 2013. URL http://arxiv.org/abs/1311.2901.

8

