ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks

Tae-Hoon Kim 1 Jonghyun Choi 2

arXiv:1801.00904v2 [cs.CV] 20 Feb 2018

Abstract
We propose to learn a curriculum or a syllabus for
supervised learning and deep reinforcement learning with deep neural networks by an attachable
deep neural network, called ScreenerNet. Specifically, we learn a weight for each sample by jointly
training the ScreenerNet and the main network in
an end-to-end self-paced fashion. The ScreenerNet neither has sampling bias nor requires to
remember the past learning history. We show
the networks augmented with the ScreenerNet
achieve early convergence with better accuracy
than the state-of-the-art curricular learning methods in extensive experiments using three popular
vision datasets such as MNIST, CIFAR10 and Pascal VOC2012, and a Cart-pole task using Deep
Q-learning. Moreover, the ScreenerNet can extend other curriculum learning methods such as
Prioritized Experience Replay (PER) for further
accuracy improvement.

1. Introduction
Training a machine learning model with chosen training
samples in a certain order improves the speed of learning
and is called Curriculum Learning (Bengio et al., 2010).
The curriculum learning recently gains much attention due
to the difficulty of training deep models for reinforcement
learning (Andrychowicz et al., 2017; Graves et al., 2017;
Sukhbaatar et al., 2017). However, selecting and ordering
samples is a hard-decision process and significantly lowers
the chance of samples to be selected in the later iterations
and changes solution to which the model converges if the
samples are rejected earlier (sampling bias). In addition, the
decision criteria are mostly defined by a set of hand-crafted
rules such as classification error or confidence of the main
network in the majority of previous work (Bengio et al.,
2015; 2009; Graves et al., 2017). Those hand-crafted rules
require additional rules to handle the sampling bias.
To alleviate the sampling bias, we present a scheme to determine a soft decision by a weight value of every training
1
Intel, Korea 2 Allen Institute for Artificial Intelligence
(AI2), Seattle, WA, USA. Correspondence to: Jonghyun Choi
<jonghyunc@allenai.org>.

Replay

Rules

Training
Samples

✓

Discarded
Samples

Soft Curriculum
by weights
0.1

0.9

Loss

0.8

Main Network

ScreenerNet
Joint Training

Figure 1. ScreenerNet learns the valuation of training samples for
a main network. It regresses the sample into its significance for
training; significant if it is hard for the current performance of the
main network, and non-significant if it is easy.

sample for building a curriculum. Moreover the soft decision is learned in a self-paced manner (Kumar et al., 2010;
Jiang et al., 2015; 2014; Supancic & Ramanan, 2013; Ma
et al., 2017) without requiring to memorize the historic loss
values (Schaul et al., 2016). This is a generalization of the
hard decision by considering all samples at every curriculum
update, thus never ignores any samples at any iteration to
maximize the efficacy of the curricular learning, motivated
from the recent work of efficiently exploring spaces of state,
action, and goal through the learning (Andrychowicz et al.,
2017; Sukhbaatar et al., 2017; Jiang et al., 2015) and is
similar to (Hinton, 2007; Schaul et al., 2016; Jiang et al.,
2015). Moreover, to discover the rules for curriculum at
each training iteration that are beyond our intuition, we learn
the curriculum by an attachable neural network along with
the main network being trained. We call the attachable network as ScreenerNet. The ScreenerNet provides the locally
most accurate weights per the main network being trained
in a self-paced manner.
Unlike the many recent approaches that are mostly applied
to deep reinforcement learning problems (Schaul et al.,
2016; Andrychowicz et al., 2017; Fan et al., 2017), we
empirically show that our ScreenerNet can be applied to
various types of networks including convolutional neural
network for visual recognition as well as deep reinforcement
learning. The ScreenerNet not only expedites the convergence of the main network but also improves the accuracy
at the end of the training. Finally, the ScreenerNet can
extend the stochastic sampling based curriculum learning
approaches for further improvement of final accuracy.

ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks

2. Related Work
As a pioneering work of curriculum learning, Hinton (Hinton, 2007) introduced an error-based non-uniform sampling
scheme by importance sampling to significantly speed-up
training of a network on a digit classification using MNIST
dataset. The idea of error-based correction has been widely
used as one of the most popular cues in the curriculum learning literature. Since Bengio et al. (Bengio et al., 2009) coin
the term curriculum learning for the method, number of
approaches (Graves et al., 2017; Lee & Grauman, 2011;
Schaul et al., 2016; Lin et al., 2017; Zhou et al., 2017) have
been proposed to sample, weigh, or sort the training examples to improve accuracy and/or expedite the training.
However, these methods require to design a hand-crafted
rule and/or to run the training procedure twice to obtain
accurate estimate of sample valuation. To address the issues,
Jiang et al. propose to estimate the sample valuation at the
same time of training a learning model (Jiang et al., 2015)
by adding a term in the objective function. This is an extension of Kumar et al.’s work on Self-Paced learning (Kumar
et al., 2010) in the curriculum learning literature.
Recently, Schaul et al. (Schaul et al., 2016) proposed
a sampling scheme to increase the replay probability
of training samples that have a high expected learning
progress determined by an optimization loss. They also
proposed a weighted importance sampling method to address the bias of sampling, which is similar to our weighting
scheme. In stochastic gradient-based optimization perspective, Loshchilov and Hutter (Loshchilov & Hutter, 2016)
also proposed to sample mini-batches non-uniformly, called
AdaDelta (Zeiler, 2012) and Adam (Kingma & Ba, 2014).
Most recently, Graves et al. (Graves et al., 2017) proposed
an automatic curriculum learning method for LSTM for
the NLP application. They define a stochastic syllabus by
a non-stationary multi-armed bandit algorithm of getting
a reward signal from each training sample. They defined
mappings from the rates of increase in prediction accuracy
and network complexity to the reward signal.
Koh and Liang (Koh & Liang, 2017) adopted influence
functions from robust statistics to measure the effect of parameter changes or perturbations of the training data such as
pixel values. It was applied to debugging models, detecting
dataset error, and training-set attack. Although the direct
application to curriculum learning is not presented, the authors show a way of predicting the significance of training
samples, which can be useful for learning curriculum.
Similar to our idea of attaching another network to the main
one, self-play between the policy and a task-setting was
proposed by Sukhbaatar et al. (Sukhbaatar et al., 2017). The
task-setter tries to find the simplest tasks that the policy
cannot complete, which makes learning for the policy easy

because the selected task is likely to be only just beyond
the capability of the policy. Similarly, Andrychowicz et
al. (Andrychowicz et al., 2017) recently proposed to learn a
curriculum but with a hindsight replay. They use unshaped
reward signals even when they are sparse and binary, without requiring domain knowledge. These recent approaches
address the sparsity and complexity of solution spaces in
deep reinforcement learning, which is not a main issue for
the supervised learning.
Very recently, Zhou et al. (Zhou et al., 2017) proposed
an adaptive feeding method that classifies an input sample
as easy or hard one in order to forward the input to an
appropriate one of a fast but less accurate neural network and
an accurate but slow network for object detection. However,
they only showed the speed up the inference preserving the
accuracy of the main classifier.
In addition, Jiang et al. (Jiang et al., 2017) propose an attachable network to regularize the network not to overfit to
the data with corrupted labels, called MentorNet. The MentorNet uses feature embedding of the sample by the main
network, referred as StudentNet, and compute a weight for
each sample. It is pretrained using a few steps of loss history
of the StudentNet with LSTM and is attached to the StudentNet while its weights are fixed due to the risk of trivial
solution. Whereas, focusing on faster and better convergence of the main network, our ScreenerNet is an attachable
network that is jointly trained with the main network in an
end-to-end fashion with no pretraining required and with
less chance of falling into a trivial solution.
Unlike the previous approaches, our ScreenerNet has two
benefits: First, it includes all samples to update weight even
though the samples have weight values close to zero. This
benefit is particularly significant in stochastic sampling approaches, if an important sample is assigned a low sampling
priority at the early stage of training, it may not be likely to
be picked again and may not be significantly used for the
training until other samples have low sampling priority as
well. Second, it can learn a direct mapping of a training
sample to its significance, even if the training sample is
unseen, unlike the other memory-based methods.
To the best of our knowledge, ScreenerNet is the first approach of an attachable deep weight regressor to both speed
up the training and improve the accuracy without modifying
the main network. Moreover, it can extend the stochastic
sampling curriculum learning approaches to improve further
by taking the benefits of both methods.

3. Approach
We formulate the problem of building a curriculum by learning the importance of each sample for training the main
network by a scalar weight in a joint learning framework.

ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks

Specifically, we define the online importance of a training
sample x as a random variable wx . Since we want to train
the main network better in speed and accuracy, the objective of the curriculum learner maximizes the likelihood by
sample weight wx given the error E between the prediction
of the main network for x and its target. The maximum
cx can be written as follows:
likelihood estimator of wx , w
cx = arg max P (E|wx , Wc )
w
wx

0.8
0.6

wx

= arg max

1

P (wx |E, Wc )P (E|Wc )
,
P (wx |Wc )

(1)

0.4
0.2

where Wc is a parameter of the main network. Since E is a
function of Wc , Equation 1 can be reduced to

0

0
1

0.5

Figure 2. Loss function0.5LS (ex , wx ) with M = 1 without the L1
1 our experiments.
0 for
regularizer. We bound wxwto be in (0, 1)
ex
x

P (wx |E)P (E|Wc )
arg max
.
wx
P (wx |Wc )

(2)

to increase weight wx when error of the main network ex is
high and promotes to decrease weight when the error is low.

The ScreenerNet is a neural network to optimize the objective.

Note that we bound the weight value by the ScreenerNet
to be in (0, 1) by a Sigmoid layer at the end of the ScreenerNet, since its multiplication to the gradient without the
bound may cause overshooting or undershooting of the main
network (Hinton, 2007).

3.1. ScreenerNet
Valuating the exact significance of each training sample to
maximize the final accuracy of the main network is computationally intractable (Graves et al., 2017). The influence
functions in (Koh & Liang, 2017) could be a potential solution to estimate the final accuracy with less computational
burden but still requires significant computational cost at the
initialization of every iteration of training the main network.
Instead of estimating the final accuracy, we propose to simplify the problem of sample-wise significance valuation to
a local optimal policy that predicts the weights of training
samples at the current iteration of the training.
Let wx be a weight of training sample, x, predicted by
ScreenerNet, S. Let LF (F(x), tx ) be an objective function
for the main network F(·) to compute an error between
F(x) and its target label tx . We define an objective function
that ScreenerNet minimizes, LS (·), as follows:
X
x∈X

X

(1 − wx )2 ex + wx2 max(M − ex , 0) + α
kpk1 , (3)
p∈WS

where wx = S(x), ex = LF (F(x), tx ), WS is parameters
of the ScreenerNet S, and α is a balancing hyper-parameter
for the regularizer of the ScreenerNet. X is a set of training
images and M is a margin hyper-parameter. We plot LS
with M = 1 except the L1 regularizer in Figure 2.
As shown in the Figure 2, the objective function is a nonnegative saddle like function with minima at (wx , ex ) =
(0, 0) or (1, 1) and with maxima at (wx , ex ) = (0, 1) or
(1, 0). Thus, the LS promotes the sample weight to be high
when the error (loss) of the sample by the main network is
high and vice versa. Therefore, the ScreenerNet encourages

Optimization. Optimizing the loss of the ScreenerNet augmented network is not trivial as the loss is non convex. Thus,
we employ the block-coordinate descent to optimize the network and depict the gradient path of the subset of variables
in Figure 3. The numbers in the blue boxes in the figure
denotes the order of updating the gradient.

F

ŷ

Main Network

Back-propagation

5

1

3

LF (ŷ, 1)
LF (ŷ, wx )

x
Input Image

S

ScreenerNet

2

ex
4

eweighted
x

LS

wx
7

6

e wx

Back-propagation

Figure 3. Optimization path of a ScreenerNet attached network.
The numbers in blue boxes show the order of gradient update.

By the block coordinate descent, at each iteration of the
epochs in the training phase, we predict the weight, wx of
the training sample x to update the main network F using
the error, ex from x and the weight wx . Then, we update
ScreenerNet using the error, ex . The training procedure
of ScreenerNet augmented deep network is summarized as
Algorithm 1.
Architecture of ScreenerNet. A larger ScreenerNet than
the main network would have larger capacity than the main
network, then it may require more sample to reliably pre-

ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks

Algorithm 1 Training a network with ScreenerNet
1: Given: training samples (X, tX ), main network F,
ScreenerNet S
2: Initialize F, S
3: for each iteration of the training do
4:
x ← Sample(X).
. sample a mini-batch from X
5:
wx ← S(x).
. predict an weight of a sample
6:
ŷ ← F(x).
. prediction from the main network
7:
eweighted
←
L
(ŷ,
wx ). . compute a weighted error
F
x
8:
ex ← LF (ŷ, 1).
. compute an error of the sample
9:
UpdateNetwork(F, ew
. train main network
x ).
10:
UpdateNetwork(S, ex ).
. train ScreenerNet
11: end for

dict the significance of the samples. A simple ScreenerNet
would not predict the significance well and likely predicts
it like a uniform distribution. We empirically found that
ScreenerNet whose architecture is similar to that of the
main network or slightly simpler performs the best. Further,
we also try weight sharing with the main network (refer to
Section 4.6 for details).

4. Experiments
4.1. Datasets
We have evaluated our algorithm with three popular
vision datasets; MNIST (Lecun & Cortes, 1992), CIFAR10 (Krizhevsky, 2012) and Pascal VOC 2012 (Everingham et al., 2012), and a Cart-pole example using the
deep Q-learning (van Hasselt et al., 2016; Mnih et al., 2013),
which is one of the most popular tasks in deep reinforcement
learning.
MNIST dataset has 60, 000 images with size of 28×28 in
training and 10, 000 images in testing set. CIFAR10 has
50, 000 images with size of 32×32 in training and 10, 000
images in testing set. Pascal VOC 2012 has 5, 717 images
in training and 5, 823 images in validation set. Both MNIST
and CIFAR10 are widely used for many neural network
training benchmarks. Pascal VOC 2012 is widely used
as one of the most popular visual recognition benchmarks
along with the ImageNet.
For the Cart-pole experiment, we use Cart-pole-v0 in OpenAI Gym (Brockman et al., 2016), which gets 4-dimensional
input of the state of the cart and pole, and has 2 discrete
actions to move left or right.

3.2. Extension with Stochastic Sampling
Since the ScreenerNet is a weight regression network, it
can extend the stochastic sampling approaches such as Prioritized Expereience Replay (PER) (Schaul et al., 2016).
The extension has two benefits; first, it may further improve
the convergence speed and the final accuracy than the single deployment of either methods. Second, it reduces the
computation for the ScreenerNet.
For instance, if we combine the ScreenerNet with the PER.
The PER determines the probability of a training sample to
be selected by

P (x) = P

pα
x

x̃∈X

pα
x̃

,

(4)

where px > 0 is the priority of each sample x, and α
controls how much prioritization is used. When α = 0,
it is equivalent to the uniform sampling. The priority is
defined as:
px = |ex | + ,

(5)

where  is a very small constant to prevent from assigning
zero priority to x. To extend the PER or any other sampling
based methods with the ScreenerNet, we can simply predict
weights of x that the PER or other sampling based methods
select. (Refer to the Sec. 4.4)

4.2. Experimental Set-up
We sketch the neural network architecture of the main network and the ScreenerNet used in our experiments on each
dataset and their important hyper-parameters. The detailed
architectures and set-ups are summarized in the supplementary material.
MNIST. We use a simple CNN architecture (consisting of
two convolutional layers, two fully connected layers) for
main network and use a simpler ScreenerNet than the main
network (refer to the dicussion in Sec.4.6). We use minibatches of 64 samples, α = 0.01, and the margin M = 1.0,
respectively.
CIFAR10. We have three architectures for main network;
first, a simple CNN adopted from the PyTorch tutorial, second, its variant with more convolutional layers and last, the
VGG-11 (denoted as ‘small’, ‘mid’ and ‘VGG11’ in Figure 4, respectively). We show the three set-ups to compare
the gain by the ScreenerNet when complexity of the main
network varies. For the loss fuction of the main network,
we used the negative log-likelihood. We use mini-batches of
64 samples and α = 0.01 in Equation 3 for all experiments.
For the margin M , we linearly annealed it from 20 to 5
through 50 epochs, since the error of the main network was
large at the early training iterations.
Pascal VOC 2012. We fine-tune a pre-trained VGG-19 (Simonyan & Zisserman, 2015) by replacing its final layer with
a few layers for both main network and ScreenerNet. We

ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks

98.0%

20

40

60

Epoch

80

90.0%

70.0%
60.0%
Baseline: small
Baseline: mid
Baseline: VGG11

50.0%

Baseline
ScreenerNet
100

40.0%

190
185

Accuracy

Accuracy

Accuracy

98.5%

Cartpole

VOC2012 classification

80.0%

99.0%

97.5%

CIFAR10 classification

90.0%

10

20

ScreenerNet: small
ScreenerNet: mid
ScreenerNet: VGG11
30

40

Epoch

Average Reward

MNIST classification

99.5%

80.0%
70.0%

Baseline (train)
Baseline (test)

60.0%
50.0%

10

20

30

Epoch

40

50

175
170
165

ScreenerNet (train)
ScreenerNet (test)

50

180

Baseline (Double DQN)
DDQN + ScreenerNet

160
60

10000

20000

30000

40000

50000

Steps

(a) MNIST
(b) CIFAR10
(c) Pascal VOC 2012
(d) Cart-pole
Figure 4. Comparison of the test accuracy (except (c) showing both the training and test accuracy) of main network (Blue) and the
ScreenerNet augmented network (Red)

used a binary cross entropy loss for multi-label one-vs-all
classification, mini-batches of 32 samples, the margin of
M = 1.0, and α = 0.001. For the evaluation metric, we use
the mean Average Precision (mAP) across all the 20 classes.
We apply data augmentations including random horizontal
flip, rotation and cropping.

In the results for the MINST dataset shown in Figure 4(a), both the baseline and ScreenerNet augmented networks
converge quickly and the learning curves at the early stage
of training are similar. But ScreenerNet improves the final
accuracy though the gain is marginal. We discuss the results
in the second next paragraph.

Cart-pole. We designed a simple CNN for a deep Qnetwork of main network and use the same architecture
for ScreenerNet. We used α = 0.001 and the margin of
M = 2.0. We limit the maximum reward from an episode
to be 200, at which the episode finished.

In the results for the CIFAR10 dataset shown in Figure 4(b), with the simple main network architecture (shown in
tinted solid line), we observed that ScreenerNet yielded
a significant improvement in the learning speed and final
accuracy. With the mid-size main network, the ScreenerNet
shows decent improvement. But with the pretty complicated
main network (VGG11), the ScreenerNet either improves
marginally or performs on par.

We also compare the ScreenerNet with the state-of-the-art
stochastic sampling based curriculum learning method, Prioritized Experience Replay (PER) (Schaul et al., 2016).
For the PER, we used importance-sampling weights wi =

β
1
1
, where β is linearly annealed from 0.4 to 1.0
N · P (i)
until 40, 000 steps then is fixed to 1.0. We set the discount
parameter γ to be 0.99 and the replay memory to be a sliding window memory of size 50, 000. The PER processes
mini-batches of 32 samples (for visual recognition task) or
32 transitions (deep Q-learning) sampled from the memory. In Cart-pole task, a mini-batch is updated at every 32
transitions entering the memory. We use the epsilon greedy
exploration that linearly anneals epsilon value from 1.0 to
0.1 in the first 1, 000 steps. For the evaluation at every
5, 000 step, we disabled the exploration to prevent random
behavior. The evaluation is repeated 100 times to get the
final average reward. The main network for the Double
DQN is interpolated with a smoothness parameter value of
0.01 at every training iteration.
Our evaluation metric is two folds; speed of convergence
in training and the final classification accuracy for the visual recognition tasks (MNIST, CIFAR10, and Pascal VOC
2012) and the average reward per episode for the Cart-pole.
4.3. Faster and Better Convergence by ScreenerNet
We compare the test accuracy curves of the baseline network (main network only) and the ScreenerNet augmented
network (detnoted as ‘ScreenerNet’) in Figure 4 for all four
tasks.

With the result for the MNIST and CIFAR10 dataset, we
argue that when the main network is large enough, curriculum does not matter for the performance as they can simply
memorize all the details of all samples. Even though we
omit the training curve for visual clarity, the VGG11 quickly
overfits to the training set of the CIFAR10 dataset while others (small and mid) less overfit which is a usual scenario
for recognition; a less capable main network for large scale
datasets. Unless the main network is easily overfitting to the
dataset, the ScreenerNet shows decent improvement.
In the results for the Pascal VOC 2012 dataset shown in Figure 4-(c), ScreenerNet helps the main network to improve
the learning speed at early epochs even though VGG-19 is
pre-trained with a much larger dataset (ImageNet). ScreenerNet also improves the accuracy when the networks are
close to convergence. Note that even though we apply extensive data augmentations for training, the ScreenerNet
significantly improves the learning curve with training set
and improve the test accuarcy as well. We believe that
the ScreenerNet helps better training rather than overfitting.
(refer to discussion about Parameter Sharing in Sec.4.6)
In the results for the Cart-pole task, the average reward
obtained at every 5, 000 training step is shown in Figure 4(d). The ScreenerNet augmented network begins with higher
average reward than the baseline Double DQN (DDQN)
and clearly shows the higher overall gain comparing to the
baseline.

ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks

VOC2012 classification

98.5%

20

40

60

80

60.0%

50.0%

Baseline
ScreenerNet
PER

98.0%

Accuracy

99.0%

100

40.0%

Baseline
ScreenerNet
PER
10

20

Epoch

185

30

40

80.0%
70.0%
Baseline
ScreenerNet
PER

60.0%

50

Epoch

(a) MNIST

Cartpole
190

90.0%

70.0%

Accuracy

99.5%

Accuracy

CIFAR10 classification

80.0%

Average Reward

MNIST classification

100.0%

50.0%

10

20

30

40

50

Epoch

(b) CIFAR10

(c) Pascal VOC 2012

180
175
170
Baseline (Double DQN)
DDQN + PER
DDQN + ScreenerNet

165
160
60

10000

20000

30000

40000

50000

Cartpole

(d) Cart-pole

Figure 5. Comparison with PER (Green), Baseline (the main network only, Blue) and ScreenerNet augmented Network (Red)
190

Average Reward

185
180
175
Baseline (Double DQN)
DDQN + PER
DDQN + ScreenerNet
DDQN + PER + ScreenerNet
DDQN + ScreenerNet Sampling

170
165
160
10000

20000

30000

40000

50000

Steps

Figure 6. Learning curves for the cart-pole problem by average reward. Extension of combination of ScreenerNet and PER (Orange)
and stochastic sampling using ScreenerNet (Purple)

Compare with Prioritized Experience Replay (PER).
While the PER selects samples by a hard decision, ScreenerNet selects samples by a soft decision by a weight. They
could be both comparable and able to be combined together,
which we will discuss further in Sec. 4.4. To see the benefit
of ScreenerNet over the PER (Schaul et al., 2016), we compare ScreenerNet with PER for all four datasets in Figure 5.
We observe that ScreenerNet shows better learning of the
main network than the PER in all four evaluations.
4.4. Extension with Stochastic Sampling Methods for
Deep Q-Learning
Since the stochastic sampling methods determine whether
the samples to be included in the training or not by a defined
probability, we can extend them by the ScreenerNet to investigate the synergistic effect of both methods. Particularly, we
can extend the ScreenerNet in two ways; first, ScreenerNet
provides weights on the selected samples by the stochastic
sampling methods. Second, the sampling selects the sample
by the ScreenerNet’s weight. We evaluate both scenarios
with the Prioritized Experience Replay (PER) as a choice of
stochastic sampling methods for the deep Q-learning task
using the cart-pole task in Figure 6.
4.4.1. S CREENER N ET W EIGHS S AMPLES S ELECTED BY
THE P RIORITIZED E XPERIENCE R EPLAY (PER)
For the first scenario, we weigh the samples by the ScreenerNet on the samples selected by the PER (denoted as Screen-

erNet + PER (orange solid line) in Figure 6). This extension outperforms the PER (green dotted line) but performs
worse than the ScreenerNet (red dotted line); since ScreenerNet weighs already selected samples, ScreenerNet+PER
performance could be bounded by the performance of the
ScreenerNet.
Interestingly, the ScreenerNet(red dotted line), the PER
(green dotted line) and ScreenerNet+PER (orange solid line)
show similar learning progress; it begins with high accuracy
but falls then gradually increases. Since they all have a
sample weighting scheme, we believe that the methods with
weighting schemes quickly learn the easy solutions in the
beginning of learning (e.g., move the pole slightly from the
initial position) then learn more difficult solutions thus the
rewards drop in the early stage of learning and increase.
4.4.2. PER WITH S CREENER N ET P ROBABILITY
In the PER, the probability that determines the priority of
samples to be selected is defined by the error of the main
network. For the second scenario of the ScreenerNet extension, we replace the probablity px in Equation 5 with the
output of ScreenerNet (px = S(x) + ) and setting α = 1
in Equation 4.
The result is shown by the purple solid line
(DDQN+ScreenerNet Sampling) in Figure 6. Comparing the PER (green dotted line), the PER outperforms
the ScreenerNet Sampling in the beginning of training
where ScreenerNet is not trained enough but eventually the
ScreenerNet probability outperforms the direct error value.
4.5. Qualitative Analysis
Error Analysis with MNIST. To better understand the effect of the ScreenerNet, we investigate failure cases of
ScreenerNet augmented network (ScreenerNet) and the
main network alone (baseline) evaluated for the MNIST
dataset. We present confusion matrices of the baseline and
ScreenerNet in Figure 7 only with failed examples for visual
clarity. We also show qualtitative comparison that the failure cases that either only the ScreenerNet or the baseline
classifies incorrectly in Figure 8.
It is observed that the widely spread confusions of the base-

ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks

0

1

0

0

0

0

0

2

0

0

3

0

0

1

0

0

4

0

1

2

4

0

0

0

0

0

0

1

0

1

5

1

0

0

5

0

0

2

1

6

4

2

0

0

1

4

0

7

0

3

9

0

1

0

0

8

3

9

2

2

1

1

2

0

1

0

0

0

0

1

0

0

1

0

0

1

0

0

0

4

0

0

7

7

2

0

0

0

0

0

0

1

3

0

0

6

0

6

3

0

0

0

0

0

2

0

3

2

0

4

5

4

0

0

1

0

0

0

2

0

1

8

0

0

4

5

1

0

0

6

0

0

2

1

0

0

0

0

0

6

2

0

0

1

1

0

0

5

0

3

3

2

0

0

3

7

0

2

6

0

0

0

0

0

0

3

2

2

8

2

0

2

1

0

0

0

1

0

2

9

1

1

0

2

0

2
1

2

0

4

6

2

0

5

2

0

9

8

7

6

5

4

3

2

1

0

1.0
Ground Truth

5

5

3

5

0.6

4

0.4

1
0

2

4

0

4

2

0

0.2

0

Predicted label

Predicted label

0.0

Figure 7. Confusion matrix only with the failed examples for visual clarity. Each number in the cell shows the number of misclassification. Note that the diagonal of the matrix is zero because
success cases are excluded for the visualization. (Left) by the
network without ScreenerNet (Right) by ScreenerNet augmented
network.

995 779 887 660 771 993 006 880 553 998

622 866 944 599 355 311 599

994 995 335 991 665 993 550 774 993 997

711 655 944 866 899 866 944

221 665 335 994 994 990 884 006 772 449

733 055 944 644 355 244 944

993 997 006 002 772 885 772 114 885 556

977 599 599 100 722

(a) Baseline only fails

5

0.8

wx

2

0

1

0

8

9

1

0

8

0

0

7

0

2

6

1

0

5

3

0

4

0

1

3

0

0

8

0

2

0

1

Confusion matrix of ScreenerNet
0

1

2

0

True label

0

0

9

0

0

0

True label

Confusion matrix of Baseline
0

(b) ScreenerNet only fails

Figure 8. Comparison of exclusive failure examples. (a) baseline
only fails but ScreenerNet succeeds and (b) ScreenerNet only fails
but baseline succeeds. The three numbers under each sample image
denote classification from ScreenerNet (in red), ground-truth (in
green), and the baseline (in blue), respectively from left to right.

line are reduced overall by ScreenerNet in Figure 7. But it
also has a few new failures like mis-classification of 4 into 9.
Note that the failure examples of 4 into 9 (Figure 8-(b)’s 944
examples) are extremely challenging. Instead, ScreenerNet
increases the precision of recognizing 4, being more strict
with classifying 1, 7, 8, and 9 into 4. Similarly ScreenerNet
increases recall for 8 at the expense of reduced precision.
Easy and Difficult Training Samples Selected by ScreenerNet. As the learning of ScreenerNet augmented neural
network proceeds, difficult samples should receive high attention in the training procedure by high weight, while easy
samples should receive low attention by low weight. We
plot the tracked weight of some easy and difficult samples
as the learning proceeds in Figure 9-(a) and also present the
samples with highest and lowest weights at the end of the
training in Figure 9-(b) and (c).
The samples with high weights are visually difficult to distinguish, while the ones with low weights are visually distinctive to the other class thus training with these in the later
epoch would not add much value to improve the accuracy.

20

40

60

80

100

Epoch

(a) Weight progress of MNIST

5

3

3

3

3

3

3

2

4

6

0.649 0.544 0.542 0.537 0.522

0.010 0.012 0.012 0.012 0.012

(b) Ends with highest weights

(c) Ends with lowest weights

Figure 9. Weight progress curves and their corresponding training
images in MNIST. In (b) and (c) Green number denotes the ground
truth label and the number below denotes the final weight value.

4.6. Choice of Architecture of ScreenerNet
Complexity of the ScreenerNet. In our experiments, we
use ScreenerNet architectures that are identical to the main
network or slightly simpler than that. An intution behind this
choice is that the ScreenerNet architecture resembling the
main network is expected to have output distribution similar
to that of the main network. Thus, ScreenerNet might serve
as a conjugate prior for the maximum a posteriori estimator
(ScreenerNet augmented network) that outputs the same
distribution of the main network itself.
Empirically, we compare the learning curves of the variously
complicated ScreenerNet architectures in Figure 10-(a); the
simple (Simple SNet), the complex (Complex SNet) and the
one similar to the main network (ScreenerNet). The simple
architecture consists of two convolutional, two maxpool
with ReLU activation and one fully connected layers with a
sigmoid activation. The complex architecture consists of the
pre-trained VGG-19 followed by two fully connected with a
sigmoid activation. We use CIFAR10 dataset for evaluation.
The Simple SNet performs worse than the baseline (the
main network only); it begins with the accuracy as high as
the ScreenerNet but ends up with the lower accuracy than
the baseline. The Complex SNet performs even worse from
the beginning. We argue that the Simple SNet performs
well at the early stage of the training because it is simple
enough to quickly model the behavior of the main network.
In the later epochs, however, the Simple SNet is not strong
enough to learn details of the behavior. Whereas the Complex SNet is expected to be strong to learn the behavior,
but it is not easy for it to quickly catch up predicting its
behavior of the main network and eventually falling into

ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks

of mini-batches to update ScreenerNet while weight prediction is done for all training samples. For example, choosing
one of every two samples to update ScreenerNet reduced
around 10−20% of the computation time in our experiment,
with the acceptable degradation of the accuracy. Alternative
approach is to combine PER to ScreenerNet at the expense
of accuracy as discussed in Sec. 4.4.

CIFAR10 classification

80.0%
70.0%

Accuracy

60.0%
50.0%
40.0%
30.0%

Baseline (train)

Simple SNet (train)

Baseline (test)

Complex SNet (train)

ScreenerNet (train)

Simple SNet (test)

ScreenerNet (test)
20.0%

10

20

Complex SNet (test)
30

40

5. Conclusion
50

Epoch

We propose to estimate the significance of the training samples for effective curriculum learning by augmenting a deep
neural network, called ScreenerNet, to the main network and
jointly train them. We demonstrated that the ScreenerNet
helps training deep neural networks both by fast and better
convergence in various tasks including visual classification
and deep reinforcement learning such as deep Q-learning.
Moreover, the ScreenerNet not only outperforms the stateof-the-art sampling based curriculum learning method such
as the Prioritized Experience Replay (PER) (Figure 5) but
also can extend it for further improvement (Figure 6).

(a) Different complexity of ScreenerNet’s.
CIFAR10 classification for shared parameters
70.0%

Accuracy

60.0%
Baseline (test)
50.0%

ScreenerNet (test)
Parameter sharing 1 (train)
Parameter sharing 1 (test)

40.0%

Parameter sharing 2 (train)
Parameter sharing 2 (test)
10

20

30

40

50

Epoch

(b) Parameter of ScreenerNet is shared with the main network.
Figure 10. Accuracies of different ScreenerNet configurations on
CIFAR10 dataset (small architecture).

a bad local minimum. More iterations of optimization or
higher learning rate were helpful but marginally according
to our experiments. The ScreenerNet architecture similar to
the main network empirically performs the best.
Parameter Sharing. We also tried the network parameter
sharing between common layers of the main network and
the ScreenerNet using CIFAR10 dataset. There are two
parameter sharing scenarios; first, ScreenerNet does not
change the parameter of shared layers but fine-tunes the last
FC 1 layer. Second, ScreenerNet also updates the parameter
of the shared layers.
As illustrated in Figure 10-(b), parameter sharing decreases
the accuracy. It implies that as the ScreenerNet does not
learn the same objective that the main network learns, rather
it learns the behavior of the main network; even the low
level signal to the ScreenerNet is different from that of the
main network. We argue that the ScreenerNet helps the
main network to learn faster and better not helps it overfit.
4.7. Computational Cost
ScreenerNet needs to update its weights during training at
the cost of additional computation and memory. Detailed
cost comparison can be found in the supplementary material.
To alleviate the computational burden, one can use a subset

The learning objective of ScreenerNet is not the same as
that of the main network as other work is trying to model.
Instead, the ScreenerNet estimates the probability that the
main network will correctly classify the given sample or
not. The architecture of the ScreenerNet is slightly simpler
than the target network and thus can be trained ahead of it
in terms of training maturity, which we empirically found
to perform the best.
Since the ScreenerNet is not a memory-based model, it can
also be considered as an error estimator of the current state
of the main network for the new sample. Thus, the ScreenerNet can estimate the sample confidence at inference time,
which is particularly useful for the real world reinforcement
learning system, similarly to the adaptive classifier proposed
by (Zhou et al., 2017).
Limitation. Since the objective function of the ScreenerNet
views training samples with large errors as significant ones
to train the main network, it possibly boosts weight values of
mislabeled training samples which may perturb a decision
boundary of the main network.
Future work. We can extend the ScreenerNet to be progressively expanding its complexity from a simple network
to a complex one by adding new layers as the training progresses, similarly to (Wang et al., 2017). Since the newly
added layers to the ScreenerNet might lead the learning to
be unstable, we can use a momentum in updating the weight
as wx = λSold (x) + (1 − λ)Snew (x), where Sold and Snew
are respectively previous and current networks, and λ is a
hyper-parameter in [0, 1].

ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks

References
Andrychowicz, Marcin, Wolski, Filip, Ray, Alex, Schneider,
Jonas, Fong, Rachel, Welinder, Peter, McGrew, Bob, Tobin, Josh, Abbeel, Pieter, and Zaremba, Wojciech. Hindsight Experience Replay . CoRR, abs/1707.01495, 2017.
Bengio, S., Weston, J., and Grangier, D. Label Embedding
Trees for Large Multi-Class Tasks. In NIPS, 2010.
Bengio, S., Vinyals, O., Jaitly, N., and Shazeer, N. Scheduled sampling for sequence prediction with recurrent neural networks. In NIPS, 2015.
Bengio, Y., Louradour, Jérôme, Collobert, R., and Weston,
J. Curriculum Learning. In ICML, 2009.
Brockman, Greg, Cheung, Vicki, Pettersson, Ludwig,
Schneider, Jonas, Schulman, John, Tang, Jie, and
Zaremba, Wojciech. OpenAI Gym, 2016.

Kumar, M. Pawan, Packer, B., and Koller, D. Self-Paced
Learning for Latent Variable Models. In NIPS, 2010.
Lecun, Y. and Cortes, C. The MNIST Database of Handwritten Digits. NYU, 1992.
Lee, Y. J. and Grauman, K. Learning the Easy Things First:
Self-Paced Visual Category Discovery. In CVPR, 2011.
Lin, Tsung-Yi, Goyal, P., Girshick, R., He, K., and Dollar, P.
Focal Loss for Dense Object Detection. In CVPR, 2017.
Loshchilov, I. and Hutter, F. Online Batch Selection for
Faster Training of Neural Networks. In Proceedings of
ICLR Workshop, 2016.
Ma, F., Meng, D., Xie, Q., Li, Z., and Dong, X. Self-Paced
Co-training. In ICML, 2017.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A.,
Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing
Atari with Deep Reinforcement Learning. In NIPS Deep
Learning Workshop, 2013.

Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J.,
and Zisserman, A. The PASCAL Visual Object Classes
Challenge 2012 (VOC2012) Results. http://www.pascalnetwork.org/challenges/VOC/voc2012/workshop/index.html, Schaul, Tom, Quan, John, Antonoglou, Ioannis, and Silver,
David. Prioritized experience replay. In Proceedings of
2012.
ICLR, 2016.
Fan, Y., Tian, F., Qin, T., Bian, J., and Liu, Tie-Yan. LearnSimonyan, K. and Zisserman, A. Very Deep Convolutional
ing What Data to Learn. arXiv preprint:1702.08635,
Networks for Large-Scale Image Recognition. In ICLR,
2017.
2015.
Graves, Alex, Bellemare, Marc G., Menick, Jacob, Munos,
Sukhbaatar, Sainbayar, Kostrikov, Ilya, Szlam, Arthur, and
Rémi, and Kavukcuoglu, Koray. Automated Curriculum
Fergus, Rob. Intrinsic Motivation and Automatic CurricLearning for Neural Networks. In Proceedings of ICML,
ula via Asymmetric Self-Play. CoRR, abs/1703.05407,
2017.
2017.
Hinton, G. E. To Recognize Shapes, First Learn to Generate
Supancic, J. S. and Ramanan, D. Self-paced learning for
Images. Progress in brain research, 165:535–547, 2007.
long-term tracking. In CVPR, 2013.
Jiang, L., Meng, D., Yu, Shoou-I, Lan, Z., Shan, S., and
van Hasselt, Hado, Guez, Arthur, and Silver, David. Deep
Hauptmann, A. G. Self-Paced Learning with Diversity.
Reinforcement Learning with Double Qlearning. In AAAI,
In NIPS, 2014.
2016.
Jiang, L., Meng, D., Zhao, Q., Shan, S., and Hauptmann,
Wang, Yu-Xiong, Ramanan, D., and Hebert, M. Growing
A. G. Self-Paced Curriculum Learning. In AAAI, 2015.
a Brain: Fine-Tuning by Increasing Model Capacity. In
CVPR,
2017.
Jiang, L., Zhou, Z., Leung, T., Li, Li-Jia, and Fei-Fei, L.
MentorNet: Regularizing Very Deep Neural Networks on
Corrupted Labels. arXiv preprint:1712.05055, 2017.

Zeiler, Matthew D. Adadelta: An adaptive learning rate
method. arXiv preprint:1212.5701, 2012.

Kingma, D. P. and Ba, J. Adam: A Method for Stochastic
Optimization. CoRR, abs/1412.6980, 2014.

Zhou, Hong-Yu, Gao, Bin-Bin, and Wu, Jianxin. Adaptive feeding: Achieving fast and accurate detections by
adaptively combining object detectors. In Proceedings of
ICCV, 2017.

Koh, P. W. and Liang, P. Understanding Black-box Predictions via Influence Functions. arXiv preprint:1703.04730,
2017.
Krizhevsky, Alex. Learning multiple layers of features from
tiny images. University of Toronto, 2012.

ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks

A. Architecture of the main network and the
ScreenerNet
For all experiments, we used the same or slightly simpler
network architecture for the ScreenerNetḞor the description of the neural network architectures, we use following
notations: Convk m denotes a convolution layer of k × k
kernel, m output channels, FC m denotes a fully connected
layer of which output dimension is m and MaxPool denotes
2x2 max pooling. Dropout probability is set to 0.5 for all
experiments unless mentioned. Table 1 shows the detailed
architecutres for the main network and the ScreenerNet for
all experiments. Table 2 shows the detailed optimization
parameter sets for all experiments.

Table 3. Computational cost and sizes of networks.
Cost
Main Network
ScreenerNet
Common

# of param.
ms. / sample
# of param.
ms. / sample
batch-size

CIFAR10
62,018
0.32
123,271
0.69
64

MNIST
21,840
0.37
28.017
0.62
64

VOC2012
140,097,492
13.6
279,671,829
26.9
32

Table 1. Architectures of baseline and ScreenerNet
Network
Main Network
MNIST
ScreenerNet
CIFAR10
small

CIFAR10
mid

CIFAR10
VGG11

Pascal
VOC2012

Main Network
ScreenerNet
Main Network
ScreenerNet
Baseline
ScreenerNet
Main Network
ScreenerNet
Main Network

CartPole

ScreenerNet

Architecture
Conv5 10–ReLU–MaxPool–Conv5 20–Dropout–ReLU
–MaxPool–FC 50–ReLU–Dropout–FC 10–LogSoftmax
Conv3 4–ELU–Conv3 8–ELU–Conv3 16–ELU
–Conv3 32–ELU–FC 1
Conv5 6–ReLU–MaxPool–Conv5 16–ReLU–MaxPool
–FC 120–ReLU–FC 84–ReLU–FC 10–LogSoftmax
Conv5 6–ReLU–MaxPool–Conv5 16–ReLU–MaxPool
–FC 120–ReLU–FC 84–ReLU–FC 1–Sigmoid
Conv5 32–ReLU–MaxPool–Conv5 32–ReLU–MaxPool
–FC 400–ReLU–FC 200–ReLU–FC 10–LogSoftmax
Conv5 32–ReLU–MaxPool–Conv5 32–ReLU–MaxPool
–FC 400–ReLU–FC 200–ReLU–FC 1–Sigmoid
[VGG-11 convolutional features network]–Dropout–FC 512
–ReLU–Dropout–FC 512–ReLU–FC 10–LogSoftmax
[VGG-11 convolutional features network]–Dropout–FC 512
–ReLU–Dropout–FC 512–ReLU–FC 1–Sigmoid
[VGG-19 up to FC 4096]–Dropout–FC 128–
BatchNorm–ReLU–Dropout–FC 20–Sigmoid
[VGG-19 up to FC 4096]–FC 64–ReLU–FC 1
–Sigmoid
FC 16–ReLU–FC 16–ReLU–FC 16–ReLU–FC 2
FC 16–ReLU–FC 16–ReLU–FC 16–ReLU–FC 1
–Sigmoid

Table 2. Parameters used for our experiments: optimizer algorithm,
learning rate, batch-size, margin, and regularizer in Equation (3).
Parameters
MMNIST
CIFAR10
(all)
Pascal
VOC2012
CartPole

Main Network
ScreenerNet
Main Network
ScreenerNet
Main Network
ScreenerNet
Main Network
ScreenerNet

Optimizer
SGD
Adam
SGD
Adam
SGD
Adam
Adam
Adam

LR
0.01
0.0001
0.01
0.001
0.001
0.0025
0.001
0.0015

Batch
64
64
32
32

M

α
N/A
1.0
0.01
N/A
20 to 5 0.01
N/A
1.0
0.001
N/A
2.0
0.001

B. Computational Cost
Table 3 shows the comparison of the baselind and its combination with ScreenerNet in terms of required computation
and network size. The training performance is measured for
PyTorch implementations on a single NVIDIA GTX 1080.

C. More Qualitative Results for the Error
Analysis with MNIST
We show the examples that the main network only and the
ScreenerNet augmented network both fail in Figure 11.

727 060 535 282 353 949 535 989 565 646 272

794

272 494 171 757 979 787 737 383 494 353 848

865

949 161 090 202 191 353 161 080 838 797 979

732

060 464 989 272 272 727 717 717 080 353 838

712

898 171 707 282 949 272 272 797 797 656

860

Figure 11. Examples that both ScreenerNet and baseline fail. Left:
Both ScreenerNet and baseline yield the same classification. Right:
ScreenerNet and baseline classify differently.

