1

Incremental Learning Through Deep Adaptation

arXiv:1705.04228v2 [] 13 Feb 2018

Amir Rosenfeld
John K. Tsotsos
Department of Electrical Engineering and Computer Science
York University, Toronto, ON, Canada
amir@eecs.yorku.ca,tsotsos@cse.yorku.ca
Abstract—Given an existing trained neural network, it is often desirable to learn new capabilities without hindering performance of
those already learned. Existing approaches either learn sub-optimal solutions, require joint training, or incur a substantial increment in
the number of parameters for each added domain, typically as many as the original network. We propose a method called Deep
Adaptation Networks (DAN) that constrains newly learned filters to be linear combinations of existing ones. DANs precisely preserve
performance on the original domain, require a fraction (typically 13%, dependent on network architecture) of the number of parameters
compared to standard fine-tuning procedures and converge in less cycles of training to a comparable or better level of performance.
When coupled with standard network quantization techniques, we further reduce the parameter cost to around 3% of the original with
negligible or no loss in accuracy. The learned architecture can be controlled to switch between various learned representations,
enabling a single network to solve a task from multiple different domains. We conduct extensive experiments showing the effectiveness
of our method on a range of image classification tasks and explore different aspects of its behavior.
Index Terms—Incremental Learning, Transfer Learning, Domain Adaptation

F

1

I NTRODUCTION

While deep neural networks continue to show remarkable performance gains in various areas such as image
classification [1], semantic segmentation [2], object detection
[3], speech recognition [4] medical image analysis [5] - and
many more - it is still the case that typically, a separate
model needs to be trained for each new task. Given two
tasks of a totally different modality or nature, such as
predicting the next word in a sequence of words versus
predicting the class of an object in an image, it stands to
reason that each would require a different architecture or
computation. A more restricted scenario - and the one we
aim to tackle in this work - is that of learning a representation which works well on several related domains. Such a
scenario was recently coined by Rebuffi et al. [6] as multipledomain learning (MDL) to set it aside from multi-task
learning - where different tasks are to be performed on the
same domain. An example of MDL is image-classification
where the images may belong to different domains, such as
drawings, natural images, etc. In such a setting it is natural
to expect that solutions will:
1)
2)
3)
4)

Utilize the same computational pipeline
Require a modest increment in the number of required parameters for each added domain
Retain performance of already learned datasets
(avoid “catastrophic forgetting”)
Be learned incrementally, dropping the requirement
for joint training such as in cases where the training data for previously learned tasks is no longer
available.

Our goal is to enable a network to learn a set of related tasks
one by one while adhering to the above requirements. We
do so by augmenting a network learned for one task with
controller modules which utilize already learned representations for another. The parameters of the controller modules

are optimized to minimize a loss on a new task. The training
data for the original task is not required at this stage. The
network’s output on the original task data stays exactly as
it was; any number of controller modules may be added
to each layer so that a single network can simultaneously
encode multiple distinct domains, where the transition from
one domain to another can be done by setting a binary
switching variable or controlled automatically. The resultant
architecture is coined DAN, standing for Deep Adaptation
Networks. We demonstrate the effectiveness of our method
on the recently introduced Visual Decathlon Challenge [6]
whose task is to produce a classifier to work well on ten
different image classification datasets. Though adding only
13% of the number of original parameters for each newly
learned task (the specific number depends on the network
architecture), the average performance surpasses that of fine
tuning all parameters - without the negative side effects
of doubling the number of parameters and catastrophic
forgetting. In this work, we focus on image classification in
various datasets, hence in our experiments the word “task”
refers to a specific dataset/domain. The proposed method
is extensible to multi-task learning, such as a single network
that performs both image segmentation and classification,
but this paper does not pursue this.
Our main contribution is the introduction of an improved alternative to transfer learning, which is as effective
as fine-tuning all network parameters towards a new task,
precisely preserves old task performance, requires a fraction
(network dependent, typically 13%) of the cost in terms of
new weights and is able to switch between any number
of learned tasks. Experimental results verify the applicability of the method to a wide range of image-classification
datasets.
We introduce two variants of the method, a fullyparametrized version, whose merits are described above

2

Controller Module(Wl )

X

(input)

Controller Module(Wl+1 )

Original
Convolution

Original
Convolution

Controller Module(Wk )

...

Classifier 1

+

Original
Convolution

Output

Classifier 2

Dataset Decider

α

Fig. 1: Overview of proposed method. For newly learned domains, controller modules are attached to convolutions
of a base network, whose parameter are frozen. A switching variable α allows to switch the behavior of the network
between the original behaviour of the convolution and a re-parametrized one for the new domain. α can be determined
either manually or via a sub-network (“Dataset Decider”) which determines the source domain of the image, switching
accordingly between different sets of control parameters. α Also controls which of the classifiers to apply. Other layers
inremental
learning.html
(e.g, non-linearities, batch-normalization,
skip layers)
not shown for presentation purposes. We visualize one added task,
though an arbitrary number of tasks can be added.

2

Xl

2.1
+
⋅ (1 − α)

Conv(Fl ;

Fl

⋅ α

Xl−1

Conv(Fla ;

)

F

Wl

Xl−1

)

a
l

Xl−1 , α

Fig. 2: Controller modules: filters of a convolutional layer of
a base network are modified by re-combining their weights
through a controller module, where a switching α variable
can choose between the original filters Fl and newly created
Controller Module( )
Controller Module(
)
Controller Module(
)
ones Fla . We show a controller module for a single added
task, but any number of controllers can be added, with α
...
Original then being a vector
Originalinstead of scalar.
Original
Wl

Convolution

Wl+1

Convolution

Wk

Convolution

α

Multi-task Learning

In multi-task learning, the goal is to train one network to
perform several tasks simultaneously on the same input.
This is usually done by jointly training on all tasks. Such
training is advantageous in that a single representation is
used for all tasks. In addition, multiple losses are said to act
as an additional regularizer. Some examples include facial
landmark localization [7], semantic segmentation [8], 3Dreasoning [9], object and part detection [10] and others.
While all of these learn to perform different tasks on the
same dataset, the recent work of [11] explores the ability
of a single network to perform tasks on various image
classification datasets. We also aim to classify images from
multiple datasets but we propose doing so in a manner
which learns them one-by-one rather than jointly. Concurrent with our method is that of [6] which introduces
dataset-specific additional residual units. We compare to
this work in Sec 4. Our work bears some resemblance to
[12], 1where two networks are trained jointly, with additional
Classifier
“cross-stitch” units, allowing each layer from one network
to have as additional input linear combinations of outputs
+
from a lower layer Output
in another. However, our method does
not require joint training and requires significantly fewer
Classifier 2
parameters.
2.2

and one with far fewer parameters, which significantly outperforms shallow transfer learning (i.e. feature extraction)
for a comparable number of parameters. In the next section,
we review some related work. Sec. 3 details the proposed
method. In Sec. 4 we present various experiments, including
comparison to related methods, as well as exploring various
strategies on how to make our method more effective,
followed by some discussion & concluding remarks.

R ELATED W ORK

Incremental Learning

Adding a new ability to a neural net often results in
so-called “catastrophic forgetting” [13], hindering the network’s ability to perform well on old tasks. The simplest
way to overcome this is by fixing all parameters of the
network and using its penultimate layer as a feature extractor, upon which a classifier may be trained [14], [15].
While guaranteed to leave the old performance unaltered, it
is observed to yield results which are substantially inferior
to fine-tuning the entire architecture [3]. The work of Li and

3

Hoiem [16] provides a succinct taxonomy of several variants
of such methods. In addition, they propose a mechanism of
fine-tuning the entire network while making sure to preserve old-task performance by incorporating a loss function
which encourages the output of the old features to remain
constant on newly introduced data. While their method
adds a very small number of parameters for each new task,
it does not guarantee that the model retains its full ability
on the old task. Rusu et al. [17] shows new representations
can be added alongside old ones while leaving the old
task performance unaffected. However, this comes at a cost
of duplicating the number of parameters of the original
network for each added task. In Kirkpatrick et al. [18] the
learning rate of neurons is lowered if they are found to
be important to the old task. Our method fully preserves
the old representation while causing a modest increase in
the number of parameters for each added task. Recently,
Sarwar et al. [19] have proposed to train a network incrementally by sharing a subset of early layers and splitting
later ones. By construction, their method also preserves the
old representation. However, as their experiments show,
re-learning the parameters of the new network branches,
whether initialized randomly with a similar distribution or
simply copying the old ones results in worse performance
than learning without any network sharing. Our method,
while also re-utilizing existing weights, attains results which
are on average better than simple transfer learning or learning from scratch.
2.3

Network Compression

Multiple works have been published on reducing the number weights of a neural network as a means to represent it
compactly [20], [21], gain speedups [22] or avoid over-fitting
[23], using combinations of coding, quantization, pruning
and tensor decomposition. Such methods can be used in
conjunction with ours to further improve results, as we
show in Sec. 4.

3

A PPROACH

We begin with some notation. Let T be some task to be
learned. Specifically, we use a deep convolutional neural
net (DCNN) in order to learn a classifier to solve T ,
which is an image classification task. Most contemporary
DCNN’s follow a common structure: for each input x, the
DCNN computes a representation of the input by passing
it through a set of l layers φi , i ∈ 1 . . . l interleaved with
non-linearities. The initial (lower) layers of the network
are computational blocks, e.g. convolutions with optional
residual units in more recent architectures [24]. Our method
applies equally to networks with or without residual connections. At least one fully connected layer fi , i ∈ 1 . . . c is
attached to the output of the last convolutional layer. Let
ΦFN = σ(φl ) ◦ . . . σ(φ2 ) ◦ σ(φ1 ) be the composition of all
of the convolutional layers of the network N , interleaved
by non-linearities. We use an architecture where all nonlinearities σ are the same function, with no tunable parameters. Denote by ΦFN (x) the feature part of N . Similarly,
denote by ΦCN = fc ◦ . . . σ(f2 ) ◦ σ(f1 ) the classifier part

of N , i.e. the composition of all of the fully-connected layers
of N . The output of N is then simply defined as:

N (x) = ΦCN ◦ ΦFN (x)

(1)

We do not specify batch-normalization layers in the
above notation for brevity. It is possible to drop the ΦCN
term, if the network is fully convolutional, as in [2].
3.1

Adapting Representations

Assume that we are given two tasks, T1 and T2 , to be
learned, and that we have learned a base network N to solve
T1 . We assume that a good solution to T2 can be obtained
by a network with the same architecture as N but with
different parameters. We augment N so that it will be able
to solve T2 as well by attaching a controller module to each
of its convolutional layers. Each controller module uses the
existing weights of the corresponding layer of N to create
new convolutional filters adapted to the new task T2 : for
each convolutional layer φl in N , let Fl ∈ RCo ×Ci ×k×k be
the set of filters for that layer, where Co is the number of
output features, Cl the number of inputs, and k × k the
kernel size (assuming a square kernel). Denote by bl ∈ RC
the bias. Denote by F̃l ∈ RCo ×D the matrix whose rows are
flattened versions of the filters of Fl , where D = Ci · k · k ;
let f ∈ RCi ×k×k be a filter from Fl whose values are
 1


 i
i
1
f11 · · · f1k
f11 · · · f1k




i
..
..
f1 = 

,···,f = 
.
.
1
i
fkk
fkk
(2)
The flattened version of f is a row vector:
1
1
i
i
f˜ = (f11
, · · · , fkk
, · · · , · · · f11
, · · · , fkk
) ∈ RD

(3)

“Unflattening” a row vector f˜ reverts it to its tensor form
f ∈ RCi ×k×k . This way, we can write

F˜la = Wl · F̃l

(4)

where Wl ∈ RCo ×Co is a weight matrix defining linear
combinations of the flattened filters of Fl , resulting in Co
new filters. Unflattening F˜la to its original shape results in
Fla ∈ RCo ×Ci ×k×k , which we call the adapted filters of
layer φl . Using the symbol X ⊗ Y as shorthand for flatten
Y →matrix multiply by X→unflatten, we can write:

Fla = Wl ⊗ Fl

(5)

If the convolution contains a bias, we instantiate a new
weight vector bal instead of the original bl . The output of
layer φl is computed as follows: let xl be the input of φl
in the adapted network. For a given switching parameter
α ∈ {0, 1}, we set the output of the modified layer to be
the application of the switched convolution parameters and
biases:

xl+1 = [α(Wl ⊗ Fl ) + (1 − α)Fl ] ∗ xl + αbal + (1 − α)bl (6)
The above formulation is for switching between two
different behaviours - that of a base network and that of
the controller modules learned for a new task.

caltech256
CIFAR­10
SVHN
GTSR
daimler
plankton
sketch
omniglot

caltech256
CIFAR­10
SVHN
GTSR
daimler
plankton
sketch
omniglot

caltech256
CIFAR­10
SVHN
GTSR
daimler
plankton
sketch
omniglot

caltech256
CIFAR­10
SVHN
GTSR
daimler
plankton
sketch
omniglot

4

88.2

57.6

45.9

55.1

84.8

52.1

28.5

32.3

88.2

84.9

93.5

94.4

94.3

68.7

58.0

76.2

88.2

83.3

93.9

93.3

93.0

69.6

54.7

71.9

20.4

92.5

37.2

50.6

85.0

36.4

16.0

13.9

40.1

92.5

93.9

94.9

96.8

59.8

36.2

52.9

45.0

92.5

94.2

92.5

97.8

62.6

40.3

54.9

7.5

22.7

96.2

37.9

63.9

24.3

9.4

9.6

19.5

73.7

96.2

92.6

93.9

47.8

23.3

32.8

29.5

77.1

96.2

92.9

95.2

55.7

30.0

40.1

13.9

34.1

52.0

98.2

69.0

36.3

17.2

19.6

27.4

78.9

94.0

98.2

94.7

56.3

33.8

43.0

33.2

76.8

93.8

98.2

94.1

59.3

35.9

38.5

7.1

25.5

30.9

13.8

92.9

20.8

2.8

1.4

25.0

82.2

94.5

96.3

92.9

54.3

16.4

19.3

27.0

76.0

94.1

91.1

92.9

56.4

26.5

30.4

22.2

45.4

31.1

31.3

72.6

74.5

39.5

59.9

67.4

85.2

94.8

93.1

92.8

74.5

62.7

82.0

75.2

83.3

94.3

86.8

89.5

74.5

59.8

77.9

20.9

45.5

32.4

28.1

74.6

58.3

69.2

73.3

72.9

86.8

95.6

95.6

94.3

72.9

69.2

86.2

79.3

84.0

94.8

90.6

89.7

70.7

69.2

81.7

15.6

39.9

28.5

21.6

78.7

52.6

29.0

86.9

57.6

85.4

94.6

95.2

93.6

71.2

58.4

86.9

58.3

83.0

94.1

87.2

87.6

69.1

57.9

86.9

ft­last

ft­full

ft­full­bn­off

90
75
60
45
30

(a)

Fig. 3: Transferability of various datasets to each other (ft-last) fine tuning only the last layer (full) fine-tuning all layers
(ft-full-bn-off ) fine tuning all layers while disallowing batch-normalization layers’ weights to be updated. Overall, networks
tend to be more easily transferable to problems from related domains (e.g., natural / drawing). Zoom in to see numbers. It
is recommended to view this figure in color on-line.
To allow the network to perform multiple tasks, we turn
α to a vector α ∈ {0, 1}n where n is the overall number of
tasks, so that αj = 1if we want to perform the j ’th task and 0
otherwise. The output of the l layer will be then determined
similarly to Equation 6:
Xn
xl+1 =
αi (Flai ∗ xl + bil )
i=1

Flai ,

bil

Where
are the set of adapted filters / bias for the
i’th task and we define for Fla1 = Fl , b1l = bl to include the
original task.
A set of fully connected layers fia are learned from
scratch, attaching a new “head” to the network for each
new task. Throughout training & testing, the weights of F
(the filters of N ) are kept fixed and serve as basis functions
for F a . The weights of the controller modules are learned
via back-propagation given the loss function. Weights of
any batch normalization (BN) layers are either kept fixed
or learned anew. The batch-normalized output is switched
between the values of the old and new BN layers, similarly
to Eq. 6. A visualization of the resulting DAN can be seen in
Fig. 1 and a single control module in Fig. 2.

We denote a network learned for a dataset/task S as
NS . A controller learned using NS as a base network will be
denoted as DANS , where DAN stands for Deep Adaptation
Network and DANS→T means using DANS for a specific
task T . While in this work we apply the method to classification tasks it is applicable to other tasks as well.
3.2

Additional Design Choices

In the following, we mention some additional design
choices of possible ways to augment a network with controllers, as well as provide some analysis on the incurred
cost of parameters.

3.2.1 Weaker parametrization
A weaker variant of our method is one that forces the
matrices Wl to be diagonal, e.g, only scaling the output
of each filter of the original network. We call this variant
“diagonal” (referring only to scaling coefficients, such as by a
diagonal matrix) and the full variant of our method “linear”
(referring to a linear combination of filters). The diagonal
variant can be seen as a form of explicit regularization which
limits the expressive power of the learned representation.
While requiring significantly fewer parameters, it results
in poorer classification accuracy, but as will be shown
later, also outperforms regular feature-extraction for transfer
learning, especially in network compression regimes.
3.2.2 Multiple Controllers
The above description mentions one base network and one
controller network. However, any number of controller networks can be attached to a single base network, regardless
of already attached ones. In this case α is extended to onehot vector of values determined by another sub-network,
allowing each controller network to be switched on or off as
needed.
3.2.3 Parameter Cost
The number of new parameters added for each task depends
on the number of filters in each layer and the number
of parameters in the fully-connected layers. As the latter
are not reused, their parameters are fully duplicated. Let
M = Co × D be the filter dimensions for some conv. layer
φl where D = Ci × k × k . A controller module for φl
requires Co2 coefficients for Fla and an additional Co for bal .
Hence the ratio of new parameters w.r.t to the old for φl
C ×(Co +1)
o +1
= CD+1
≈ CDo . Example: for Co = Ci = 256
is Coo ×(D+1)
input and output units and a kernel size k = 5 this equals
256+1
256·52 +1 ≈ 0.04. In the final architecture we use the total

5

Net
VGG-B(S)
VGG-B(P)
DANcaltech−256
DANsketch
DANnoise
DANimagenet
DANimagenet+sketch

C-10
92.5
93.2
77.9
77.9
68.1
91.6
91.6

GTSR
98.2
99.0
93.6
93.3
90.9
97.6
97.6

SVHN
96.2
95.8
91.8
93.2
90.4
94.6
94.6

Caltech
88.2
92.6
88.2
86.9
84.6
92.2
92.2

Dped
92.9
98.7
93.8
94.0
91.3
98.7
98.7

Oglt
86.9
83.8
81.0
85.4
80.6
81.3
85.4

Plnk
74.5
73.2
63.6
69.6
61.7
72.5
72.5

Sketch
69.2
65.4
49.4
69.2
42.7
63.2
69.2

Perf.
87.32
87.71
79.91
83.7
76.29
86.46
87.76

#par
8
8
2.54
2.54
1.76
2.76
3.32

TABLE 1: Perf: top-1 accuracy (%, higher is better) on various datasets and parameter cost (#par., lower is better) for a few
baselines and several variants of our method. Rows 1,2: independent baseline performance. VGG-B: VGG [25] architecture
B. (S) - trained from scratch. (P) - pre-trained on ImageNet. Rows 3-7: (ours) controller network performance; DANsketch as
a base network outperforms DANcaltech−256 on most datasets. A controller network based on random weights (DANnoise )
works quite well given that its number of learned parameters is a fifth of the other methods. DANimagenet : controller
networks initialized from VGG-B model pretrained on ImageNet. DANimagenet+sketch : selective control network based on
both VGG-B(P) & Sketch. We color code the first, second and third highest values in each column (lowest for #par). #par:
amortized number of weights learned to achieve said performance for all tasks divided by number of tasks addressed
(lower is better).
number of weights required to adapt the convolutional layers Φl combined with a new fully-connected layer amounts
to about 13% of the original parameters. For VGG-B, this
is roughly 21%. For instance, constructing 10 classifiers
using one base network and 9 controller networks requires
(1 + 0.13 ∗ 9) · P =2.17 · P parameters where P is the
number for the base network alone, compared to 10 · P
required to train each network independently. The cost is
dependent on network architecture, for example it is higher
when applied on the VGG-B architecture. While our method
can be applied to any network with convolutional layers, if
Co ≥ D, i.e., the number of output filters is greater than
the dimension of each input filter, it would only increase the
number of parameters.
3.3

Limitations

An immediate question that arises from our formulation
concerns the expressive power of the resulting adapted
networks. We provide some analysis here on this issue.
In regular fine-tuning or learning schemes, the weights of
each layer may take on arbitrary values, depending on
the optimization scheme and training data. In contrast, the
proposed method constrains each filter to be a linear combination of the original filters in the corresponding layer.
This may, in general, become a strong limiting factor on the
performance of the network. The filters F = f1 , . . . , fk of
a layer l can form vectors by flattening their tensor form.
F defines a basis of a subspace of filters from which the
proposed method creates new ones. As our method defines
new filters by applying a linear transformation on F , the
initial values thereof can have several effects on the resulting
networks. Two important scenarios are when (a) network
expressibility and (b) network efficiency are affected. We
provide examples of when these scenarios can occur and
some experiments to demonstrate them.

3.3.1 Expressibility
One can easily construct cases where the expressive power
of a network is critically hindered by choosing to adapt it
from an inappropriate base network. For example, assume
that the inputs of a network are RGB images. Also assume

that - for some reason - all first-layer filters have coefficients
of exactly 0 in the green and blue channels. If information
in these channels is critical for the task to be learned by
the proposed method, it will clearly fail, as the information
resides in an orthogonal feature space to that represented by
the base network’s filters.
3.3.1.1 Residual Connections: The above claim does
not hold for networks with residual connections [24]. In
Residual connections the output yi of the i0 th layer is of
the following general form:

yi = F (xi ; Wi ) + xi

(7)

where xi is the input to this layer and Wi is the layer’s
parameters. This allows preserving information from the
previous layer, as a possible function permitted by this
is the identity function. If enough information from the
signal x is preserved then the next layer can still recover
from not representing it adequately in this layer. Hence, the
expressive power of the network is probably less sensitive
to the choice of basis function in a single layer. Nevertheless,
its efficiency may be compromised, as we describe next.
3.3.2 Efficiency
In some cases, the expressive power of the network is not
immediately affected by the choice of basis function but we
claim that the efficiency of the network is. We define the
efficiency of a network w.r.t to a given image patch as the
minimal number of layers (depth) required to discriminate
it from others by a single filter. In other words, the depth
of the shallowest layer where there exists some filter whose
response correlates strongly with the presence of the patch.
An example is a 3x3 gray-scale patch with 1 in all corners
and 0 otherwise. In the first layer, a filter v of values:


+1 −1 +1
v =  −1 −1 −1 
+1 −1 +1
would perfectly match this patch. On the other hand, one
could set 9 filters of the first layer to delta functions with 1
in location i, j for each i and j in 1, 2, 3. These filters span all
possible patches of 3x3 so they do not effect expressibility.
However, keeping the first layer fixed, the second layer

6

mean performance

maximal performance

100
100

98
original
channel switch
channel switch + learn
channel switch + clean start
channel switch + noise
original +learn
transposed

60

accuracy

accuracy

80

96

94
original
channel switch
channel switch + learn
channel switch + clean start
channel switch + noise
original +learn
transposed

40
92
20
0

10

20

epoch

30

40

50

90
30.0

(a)

32.5

35.0

37.5

40.0

epoch

42.5

45.0

47.5

50.0

(b)

Fig. 4: Demonstration of different possible limits of possible method on a toy example. Transferring from a network where
the first layer contains features from an orthogonal sub-space to that required for a task can result in chance accuracy.
Please see text for details.
would be required to create a filter to match p, whereas v
expressed this directly.
3.3.3

Toy Dataset

To further demonstrate the above claims, we constructed
a toy classification task. We first describe the dataset and
experiments and then discuss the results.
The task is: given an input image of size 28x28x3, predict
the length of a bar in the an image. In this dataset, the
images are restricted to be 1-pixel width bars of length 3 · k
where k ∈ {3, 4, 5, 6, 7} over a blank background and the
class is the corresponding length of the bar. The bar lengths
were set to be in increments of 3 as lesser ones caused
difficulties for the network to learn, probably due to maxpooling operations.
The dataset is composed of 1000 examples, with a random 75%/25% train/test split. The network we tested is
deliberately quite degenerate: it contains one 5x5 filter on
the first layer and 20 filters on the second, followed by two
fully connected layers of 320x50 and 50x5.
We created 3 variants of this dataset, to test different
transfer-learning scenarios on it. The dataset variants are (1)
only red horizontal bars, (2) only red vertical bars (referred
to as “transposed”) and (3) only green horizontal bars
(referred to as “new channel”). Note that we limited our
method to operate only on the first layer, which in this case
boils down to multiplying the output of the filter by a scalar
and learning a new bias.
The network was trained for 50 epochs using the Adam
[26] optimizer (SGD did not converge as well in this case)
in all scenarios. We tested seven scenarios. These involve
different combinations of the images in the dataset, initial-

ization of the first layer and which layers may or may not
be modified by the optimizer.
•
•
•
•
•

•
•

original: horizontal dataset, first filter fixed to a
horizontal green bar, learn other layers
original+learn : horizontal dataset, learn all layers
transposed : transposed dataset, first filter fixed to a
horizontal bar, learn other layers
channel switch: new channel dataset, first filter fixed
to horizontal green bar, learn other layers
channel switch + noise: new channel dataset, first
filter fixed to horizontal green bar+ Gaussian noise,
learn other layers
channel switch + clean start: new channel dataset,
learn all layers
channel switch + learn: new channel dataset, first
filter initialized to a horizontal green bar, learn all
layers.

Each experiment was repeated 20 times to account for
variability. We plot the mean accuracy on the validation set
over the 50 training epochs.
Fig. 4 summarizes the result of these experiments. The
left sub-figure shows the mean performance over the 50
epochs and the right one shows the maximal (of the 20
trials) attained for each scenario. These show that in most
cases, we can reach good accuracy, however on the average case there are drastic differences between the obtained
performance, although this is a very simple dataset. The
original scenario, where the first layer is fixed, attains
100% accuracy. Switching the channel from green to red
(“channel switch”) but keeping the first filter fixed results
in chance accuracy, as excepted: no linear combination of

7

a purely “green” filter can represent information from the
red channel. In this case, if we allow all layers to be learned
(“channel switch + learn”), indeed the network can finally
converge to a good accuracy (Fig. 4 (b)), but the average
performance over 20 trials is far from optimal (Fig. 4 (a))
- the network converges to around 50% accuracy (where
chance performance is 20%). This confirms that even for
very simple cases, bad initialization can be detrimental to
the training process.
As a “sanity test”, we make sure that randomly initializing the network and allowing all layers to be learned in
the “channel switch” case achieves 100% performance (in
both the average and best case). This is indeed the case
(“channel switch + clean start”), although we see that the
random initialization converges at first much slower than
the informed initialization of “original”. Next, we initialize
the network as in “original” but with the “channel switch”
dataset. However, we add random noise to the first layer’s
filter, then keep it fixed (“channel switch + noise”). The
optimization is able to recover from this initialization in
both the average and best case, as the initial filter is not
orthogonal to the green channel.
The “original + learn” is initialized as “original” but
allows the first layer to be modified. While on average it
initially converges slower than the “original” scenario, on
average it performs better.
Finally, the “transposed” case, where the original filter
is fixed to be orthogonal to the required shape (horizontal
bars), attains average lower performance, and near 100%
in the best case. Note that although we cannot generate an
horizontal bar by linearly transforming the vertical one, the
information is not lost and can be recovered by filters of the
second layer, due to their convolutional nature. This is done
by a weighted summation of the shifted responses of the
horizontal filter with proper coefficients. This is an example
of the network “efficiency” being reduced, i.e, delegating to
the second layer computations which could be done in the
first.
To summarize, we conclude from the experiments on this
toy dataset that:
1)

2)

3)

4)

Setting some network weights using strong domainspecific knowledge and fixing their values, learning
only other layers can lead to fast convergence to a
strong solution.
An even stronger solution can emerge if all layers
are initialized randomly, then allowed to be updated
- though convergence will be slower.
Initializing some weights to “bad” values can cause
on-average dramatically inferior performance to the
previous two cases, with occasional but rare instances of good performance.
Initializing some weights to “bad” values, fixing
them and learning the rest of the weights will totally
and irreparably prevent the network from learning.

In the next section, we test our method on richer and more
diverse datasets, less degenerate networks (e.g., more than
one filter in the first layer), and in some cases residual
networks, which all are likely to diminish effects shown on
artificial examples.

4

E XPERIMENTS ON R EAL DATASETS

We conduct several experiments to test our method and
explore different aspects of its behavior. The experimental
section is split into two parts, the first being of a more exploratory nature and the second geared toward results in a
recent multi-task learning benchmark. We use two different
basic network architectures on two (somewhat overlapping)
sets of classification benchmarks for the respective parts.
The first is VGG-B [25] . We begin by listing the datasets we
used (4.0.1), followed by establishing baselines by training a
separate model for each using a few initial networks. We
proceed to test several variants of our proposed method
(4.1) as well as testing different training schemes. Next, we
discuss methods of predicting how well a network would
fare as a base-network (4.3). We show how to discern the
domain of an input image and output a proper classification
(4.3.1) without manual choice of the control parameters α.
In the second part of our experiments we show how our
applying our method results in leading scores in the Visual
Decathlon Challenge [6], using a different, more recent
architecture. Before concluding we demonstrate some useful
properties of our method.
All Experiments were performed using the PyTorch1
framework using a single Titan-X Pascal GPU.

4.0.1 Datasets and Evaluation
The first part of our evaluation protocol resembles that of
[11]: we test our method on the following datasets: Caltech256 [27], CIFAR-10 [28], Daimler [29] (DPed), GTSR [30],
Omniglot [31], Plankton imagery data [32] (Plnk), Human
Sketch dataset [33] and SVHN [34]. All images are resized
to 64 × 64 pixels, duplicating gray-scale images so that
they have 3 channels as do the RGB ones. We whiten all
images by subtracting the mean pixel value and dividing
by the variance per channel. This is done for each dataset
separately. We select 80% for training and 20% for validation
in datasets where no fixed split is provided. We use the
B architecture described in [25], henceforth referred to as
VGG-B. It performs quite well on the various datasets when
trained from scratch (See Tab. 1. Kindly refer to [11] for a
brief description of each dataset.
As a baseline, we train networks independently on each
of the 8 datasets. All experiments in this part are done with
the Adam optimizer [26], with an initial learning rate of 1e-3
or 1e-4, dependent on a few epochs of trial on each dataset.
The learning rate is halved after each 10 epochs. Most networks converge within the first 10-20 epochs, with mostly
negligible improvements afterwards. We chose Adam for
this part due to its fast initial convergence with respect to
non-adaptive optimization methods (e.g, SGD), at the cost
of possibly lower final accuracy [35]. The top-1 accuracy (%)
is summarized in Table 1.
4.1

Controller Networks

To test our method, we trained a network on each of the
8 datasets in turn to be used as a base network for all
1. pytorch.org

8

Effect of Controller Initialization

log­loss

7
6
5

sketch

0.82

Mean Accuracy

diagonal
random
linear_approx

8

Transferability vs. Mean Accuracy

0.84

4

0.80

plankton
omniglot

CIFAR­10

caltech256

SVHN

0.78

GTSR

0.76

0

50

100

150

daimler

200

minibatch number

0.70

0.75

(a)

0.80

0.85

Transferability

0.90

0.95

(b)

Fig. 5: (a) Controller initialization schemes. Mean loss averaged over 5 experiments for different ways of initializing
controller modules, overlaid with minimal and maximal values. Random initialization performs the worst (random).
Approximating the behavior of a fine-tuned network is slightly better (linear approx) and initializing by mimicking the
base network (diagonal) performs the best (b) Predictability of a control network’s overall accuracy average over all
datasets, given its transferability measure.
Dataset

DPed

SVHN

GTSR

C-10

Oglt

Plnk

Sketch

Caltech

ft-full

60.1

60

65.8

70.9

80.4

81.6

84.2

82.3

ft-full-bn-off

61.8

64.6

66.2

72.5

78

80.2

82.5

81

ft-last

24.4

33.9

42.5

44

44.1

47

50.3

55.6

TABLE 2: Mean transfer learning performance. We show the mean top-1 accuracy (%) attained by fine-tuning a network
from each domain to all domains. Out of the datasets above, starting with Caltech-256 proves most generic as a feature
extractor (ft-last). However, fine tuning is best when initially training on the Sketch dataset (ft-full).
others. We compare this to the baseline of training on each
dataset from scratch (VGG-B(S)) or pretrained (VGG-B(P))
networks. Tab. 1 summarizes performance on all datasets
for two representative base nets: DANcaltech−256 (79.9%)
and DANsketch (83.7%). Mean performance for other base
nets are shown in Fig. 5 (a). The parameter cost (3.2.3) of
each setting is reported in the last column of the table. This
(similarly to [6] is the total number of parameters required
for a set of tasks normalized by that of a single fullyparametrized network. We also check how well a network
can perform as a base-network after it has seen ample training examples: DANimagenet is based on VGG-B pretrained
on ImageNet [36]. This improves the average performance
by a significant amount (83.7% to 86.5%). On Caltech-256
we see an improvement from 88.2% (training from scratch).
However, for both Sketch and Omniglot the performance is
in favor of DANsketch . Note these are the only two domains
of strictly unnatural images. Additionally, DANimagenet is
still slightly inferior to the non-pretrained VGG-B(S) (86.5%
vs 87.7%), though the latter is more parameter costly.
4.1.1

Multiple Base Networks

Arguably, a good base network should have features generic
enough so that a controller network can use them for a
broad range of target tasks. In practice this may not necessarily be the case. We conjecture that using a diverse set
of features, such as that learned by training on more than

one task, will provide a better basis for transfer learning. To
use two base-networks simultaneously, we implemented a
dual-controlled network by using both DANcaltech−256 and
DANsketch and attaching to them controller networks. The
outputs of the feature parts of the resulting sub-networks
were concatenated before the fully-connected layer. This
resulted in the exact same performance as DANsketch alone.
However, by using selected controller-modules per group
of tasks, we can improve the results: for each dataset the
maximally performing network (based on validation) is the
basis for the control module, i.e., we used DANimagenet
for all datasets except Omniglot and Sketch. For the latter
two we use DANsketch as a base net. We call this network
DANimagenet+sketch . At the cost of more parameters, it
boosts the mean performance to 87.76% - better than using
any single base net for controllers or training from scratch.
Since it is utilized for 9 tasks (counting ImageNet), its
parameter cost (2.76) is still quite good.
4.1.2

Starting from a Randomly Initialized Base Network

We tested how well our method can perform without any
prior knowledge, e.g., building a controller network on a
randomly initialized base network. The total number of parameters for this architecture is ˜12M. However, as 10M
have been randomly initialized and only the controller
modules and fully-connected layers have been learned, the
effective number is actually 2M. Hence its parameter cost is

9

4.2

Initialization

One question which arises is how to initialize the weights W
of a control-module. We tested several options: (1) Setting
W to an identity matrix (diagonal). This is equivalent to
the controller module starting with a state which effectively
mimics the behavior of the base network (2) Setting W
to random noise (random) (3) Training an independent
network for the new task from scratch, then set W to best
linearly approximate the new weights with the base weights
(linear approx). To find the best initialization scheme, we
trained DANsketch→caltech256 for one epoch with each and
observed the loss. Each experiment was repeated 5 times
and the results averaged. From Fig. 5(a), it is evident that
the diagonal initialization is superior, perhaps counterintuitively, there is no need to train a fully parametrized
target network. Simply starting with the behavior of the
base network and tuning it via the control modules results in
faster convergence. Hence we train controller modules with
the diagonal method. Interestingly, the residual adaptation
unit in [6] is initially similar to the diagonal configuration.
If all of the filters in their adapter unit are set to 1 (up to
normalization), the output of the adapter will be initially
the same as that of the controller unit initialized with the
identity matrix.
4.3

Transferability

How is one to choose a good network to serve as a basenetwork for others? As an indicator of the representative
power of the features of each independently trained network N , we test the performance on other datasets, using N
for fine tuning. We define the transferability of a source task
S w.r.t a target task T as the top-1 accuracy attained by finetuning N trained on S to perform on T . We test 3 different
scenarios, as follows: (1) Fine-tuning only the last layer (a.k.a
feature extraction) (ft-last); (2) Fine-tuning all layers of N (ftfull); (3) same as ft-full, but freezing the parameters of the
batch-normalization layers - this has proven beneficial in
some cases - we call this option ft-full-bn-off. The results in
Fig 3 show some interesting phenomena. First, as expected,
feature extraction (ft last) is inferior to fine-tuning the entire
network. Second, usually training from scratch is the most
beneficial option. Third, we see a distinction between natural images (Caltech-256, CIFAR-10, SVHN, GTSR, Daimler)
and unnatural ones (Sketch, Omniglot, Plankton); Plankton
images are essentially natural but seem to exhibit different
behavior than the rest.

Shifting Representations
caltech256
CIFAR-10
SVHN
GTSR
daimler
plankton
omniglot

80

Accuracy

determined to be 0.22. We summarize the results in Tab. 1.
Notably, the results of this initialization worked surprisingly
well; the mean top-1 precision attained by this network was
76.3%, slightly worse than of DANcaltech−256 (79.9%). This
is better than initializing with DANdaimler , which resulted
in a mean accuracy of 75%. This is possible because the
random values in the base network can still be linearly
combined by our method to create ones that are useful for
classification.

60

40

20

0
0.0

0.2

0.4

0.6

0.8

1.0

α

Fig. 6: Shifting Representations. Using a single base network
Nsketch , we check the method’s sensitivity to varying values
of α by varying it in the range [0, 1]. Increasing α shifts the
network away from the base representation and towards
learned tasks - gradually lowering performance on the base
task (diamonds) and improving on the learned ones (full
circles). The relatively slow decrease of the performance on
sketch (blue diamonds) and increase in that of Plankton
(blue circles) indicates a similarity between the learned
representations.

It is evident that features from the natural images are
less beneficial for the unnatural images. Interestingly, the
converse is not true: training a network starting from Sketch
or Omniglot works quite well for most datasets, both natural and unnatural. This is further shown in Tab. 2: we
calculate the mean transferability of each dataset by the
mean value of each rows of the transferability matrix from
Fig. 3. DANCaltech−256 works best for feature extraction.
However, for full fine-tuning using DANP lankton works as
the best starting point, closely followed by DANCaltech−256 .
For controller networks, the best mean accuracy attained
for a single base net trained from scratch is attained using
DANsketch (83.7%). This is close to the performance attained
by full transfer learning from the same network (84.2%, see
Tab. 2) at a fraction of the number of parameters. This is
consistent with our transferability measure. To further test
the correlation between the transferability and the performance given a specific base network, we used each dataset
as a base for control networks for all others and measured
the mean overall accuracy. The results can be seen in Fig. 5
(b).
4.3.1

A Unified Network

Finally, we test the possibility of a single network which
can both determine the domain of an image and classify it.
We train a classifier to predict from which dataset an image
originates, using the training images from the 8 datasets.
This is learned easily by the network (also VGG-B) which
rapidly converges to 99% accuracy. With this “datasetdecider”, named Ndc we augment DANsketch to set for each
input image I from any of the datasets Di the controller
value αi of DANsketch→Di to 1 if and only if Ndc deemed

10

Accuracy vs. Method

80

100

nQuantizationBits

4
6
8
16
32

80

60

Accuracy %

Accuracy %

Accuracy vs. Quantization

FEATURE EXTRACTOR
DIAGONAL
FINETUNE
LINEAR

100

40

20

60

40

20

0

0
Airc.

C100

DPed

DTD

Flwr

Dataset

GTSR

Oglt

SVHN

UCF

Airc.

(a)

C100

DPed

DTD

Flwr

Dataset

GTSR

Oglt

SVHN

UCF

(b)

Fig. 7: (a) Accuracy vs. learning method. Using only the last layer (feature extractor) performs worst. finetune: vanilla finetuning. Diagonal : our controller modules with a diagonal combination matrix. Linear: our full method. On average, our full
method outperforms vanilla fine tuning. (b) Accuracy vs. quantization: with as low as 8 bits, we see no significant effect of
network quantization on our method, showing they can be applied together.

I to originate from Di and to 0 otherwise. This produces
a network which applies to each input image the correct
controllers, classifying it within its own domain. While the
predicted values of αi are real-valued, we set the highest
one to 1 and the rest to zero.
4.4

Visual Decathlon Challenge

We now show results on the recent Visual Decathlon Challenge of [6]. The challenge introduces involves 10 different
image classification datasets: ImageNet [36]; Aircraft [37];
Cifar-100 [28]; Daimler Pedestrians [29]; Dynamic Textures
[38]; GTSR [30]; Flowers [39]; Omniglot [40]; SVHN [34]
and UCF-101 [41]. The goal is to achieve accurate classification on each dataset while retaining a small model
size, using the train/val/test splits fixed by the authors.
All images are resized so the smaller side of each image is
72 pixels. The classifier is expected to operate on images
of 64x64 pixels. Each entry in the challenge is assigned
a decathlon score, which is a function designed to highlight methods which do better than the baseline on all 10
datasets. Please refer to the challenge website for details
about the scoring and datasets: http://www.robots.ox.ac.
uk/∼vgg/decathlon/. Similarly to [6], we chose to use a
wide residual network [42] with an overall depth of 28 and
a widening factor of 4, with a stride of 2 in the convolution
at the beginning of each basic block. In what follows we
describe the challenge results, followed by some additional
experiments showing the added value of our method in
various settings. In this section we used the recent YellowFin
optimizer [43] as it required less tuning than SGD. We use
an initial learning rate factor of 0.1 and reduce it to 0.01
after 25 epochs. This is for all datasets with the exception of
ImageNet which we train for 150 epochs with SGD with an
initial learning rate of 0.1 which is reduced every 35 epochs
by a factor of 10. This is the configuration we determined
using the available validation data which was then used to

train on the validation set as well (as did the authors of
the challenge) and obtain results from the evaluation server.
Here we trained on the reduced resolution ImageNet from
scratch and used the resulting net as a base for all other
tasks. Tab. 3 summarizes our results as well as baseline
methods and the those of [6], all using a base architecture
of similar capacity. By using a significantly stronger base
architecture they obtained higher results (mean of 79.43%)
but with a parameter cost of 12, i.e., requiring 12 times the
amount of original parameters. All of the rows are copied
from [6], including their re-implementation of LWF, except
the last which shows our results. The final column of the
table shows the decathlon score. A score of 2500 reflects the
baseline resulting from finetuning the network learned on
ImageNet to each dataset independently. For the same architecture, the best results obtained by the Residual Adapters
method is slightly below ours in terms of decathlon score
and slightly above them in terms of mean performance.
However, unlike them, we avoid joint training over all of
the datasets and using dataset-dependent weight decay.

4.5

Compression and Convergence

In this section, we highlight some additional useful
properties of our method. All experiments in the following
were done using the same architecture as in the last section
but training was performed only on the training sets of the
Visual Decathlon Challenge and tested on the validation
sets. First, we check whether the effects of network compression are complementary to ours or can hinder them. Despite
the recent trend of sophisticated network compression techniques (for example [20]) we use only a simple method of
compression as a proof-of-concept, noting that using recent
compression methods will likely produce better results. We

11

Method
Scratch
Feature
Finetune
LWF
Res.
Adapt.
Res.
Adapt
(Joint)
DAN
(Ours)

#par
10
1
10
10

ImNet
59.87
59.67
59.87
59.87

Airc.
57.1
23.31
60.34
61.15

C100
75.73
63.11
82.12
82.23

DPed
91.2
80.33
92.82
92.34

DTD
37.77
45.37
55.53
58.83

GTSR
96.55
68.16
97.53
97.57

Flwr
56.3
73.69
81.41
83.05

Oglt
88.74
58.79
87.69
88.08

SVHN
96.63
43.54
96.55
96.1

UCF
43.27
26.8
51.2
50.04

mean
70.32
54.28
76.51
76.93

S
1625
544
2500
2515

2

59.67

56.68

81.2

93.88

50.85

97.05

66.24

89.62

96.13

47.45

73.88

2118

2

59.23

63.73

81.31

93.3

57.02

97.47

83.43

89.82

96.17

50.28

77.17

2643

2.17

57.74

64.12

80.07

91.3

56.54

98.46

86.05

89.67

96.77

49.38

77.01

2851

TABLE 3: Results on Visual Decathlon Challenge. Scratch: training on each task independently. Feature: using a pre-trained
network as a feature extractor. Finetune : vanilla fine tuning. Performs well but requires many parameters. Learningwithout-forgetting (LWF, [16] slightly outperforms it but with a large parameter cost. Residual adapt. [6] significantly reduce
the number of parameters. Results improve when training jointly on all task (Res.Adapt(Joint)). The proposed method
(DAN) outperforms residual adapters despite adding each task independently of the others. S is the decathlon challenge
score.
Convergence Speed

Convergence Speed

1.0000

1.0

mode

feature extractor
diagonal
finetune
linear

0.8

0.9950

0.6

0.4

mode

feature extractor
diagonal
finetune
linear

0.2

Mean Normalized Accuracy

Mean Normalized Accuracy %

0.9975

0.9925
0.9900
0.9875
0.9850
0.9825
0.9800

0

10

20

Epoch
(a)

30

40

25

30

35

Epoch

40

45

50

(b)

Fig. 8: (a) Our method (linear) converges to a high accuracy faster than fine-tuning. The weaker variant of our method
converges as fast as feature-extraction but reaches an overall higher accuracy (7 (a)). (b) zoom in on top-right of (a).
apply a simple linear quantization on the network weights,
using either 4, 6, 8, 16 or 32 bits to represent each weight,
where 32 means no quantization. We do not quantize batchnormalization coefficients. Fig. 7 (b) shows how accuracy
is affected by quantizing the coefficients of each network.
Using 8 bits results in only a marginal loss of accuracy. This
effectively means our method can be used to learn new tasks
while requiring an addition of 3.25% of the original amount
of parameters. Many maintain performance even at 6 bits
(DPed, Flowers, GTSR, Omniglot, SVHN. Next, we compare
the effect of quantization on different transfer methods:
feature extraction, fine-tuning and our method (both diagonal and linear variants). For each dataset we record
the normalized (divided by the max.) accuracy for each
method/quantization level (which is transformed into the
percentage of required parameters). This is plotted in Fig.
9. Our method requires significantly fewer parameters to
reach the same accuracy as fine-tuning. If parameter usage
is limited, the diagonal variant of our method significantly
outperforms feature extraction. Finally, we show that the

number of epochs until nearing the maximal performance is
markedly lower for our method. This can be seen in Fig. 8.
4.6

Discussion

We have observed that the proposed method converges to
a reasonably good solution faster than vanilla fine-tuning
and eventually attains slightly better performance. The increase in classification accuracy is despite the network’s
expressive power, which is limited by our construction. We
conjecture that constraining each layer to be expressed as a
linear combination of the corresponding layer in the original
network serves to regularize the space of solutions and is
beneficial when the tasks are sufficiently related to each
other. One could come up with simple examples where the
proposed method would likely fail. This might occur if tasks
are not of the same kind. For example, one task requires
counting of horizontal lines and the other requires counting
of vertical ones, and such examples are all that appear in the
training sets, then the proposed method will likely work far
worse than vanilla fine-tuning or training from scratch. We

12

ACKNOWLEDGMENTS

1.0

This research was funded by the Canada Research Chairs
program and by the Air Force Office for Scientific Research
(USA) for which the authors are grateful.

Mean Norm. Acc %

0.8

0.6

curMode

finetune
linear
diagonal
feature extractor

0.4

R EFERENCES
[1]
[2]

0.2

0.0

[3]

0 1 2 3 4 6 9 11 12 13 17 18 19 23 24 25 47 48 51 95 96102

no. Params %

Fig. 9: Mean classification accuracy (normalized, averaged
over datasets) w.r.t no. parameters. Our method achieve
better performance over baselines for a large range of parameter budgets. For very few parameters diagonal (ours)
outperforms features extraction. To obtain maximal accuracy our full method requires far fewer parameters (see
linear vs finetune).

experimented with such scenarios in Sec. 3.3. Empirically,
we did not see this happen in any of our experiments on
“real” datasets. This may simply stem from some inherent
similarity in image classification tasks which makes some
basic set of features useful for most of them. We leave the
investigation of this issue, as well as finding ways between
striking a balance between reusing features and learning
new ones as future work.

[4]

[5]

[6]
[7]
[8]
[9]

[10]
[11]

5

C ONCLUSIONS

We have presented a method for transfer learning thats
adapts an existing network to new tasks while fully preserving the existing representation. Our method matches or
outperforms vanilla fine-tuning, though requiring a fraction
of the parameters, which when combined with net compression reaches 3% of the original parameters with no loss of
accuracy. The method converges quickly to high accuracy
while being on par or outperforming other methods with
the same goal. Built into our method is the ability to easily switch the representation between the various learned
tasks, enabling a single network to perform seamlessly on
various domains. The control parameter α can be cast as
a real-valued vector, allowing a smooth transition between
representations of different tasks. An example of the effect
of such a smooth transition can be seen in Fig. 6 where α
is used to linearly interpolate between the representation
of differently learned pairs tasks, allowing one to smoothly
control transitions between different behaviors. Allowing
each added task to use a convex combination of already
existing controllers will potentially utilize controllers more
efficiently and decouple the number of controllers from the
number of tasks.

[12]
[13]
[14]
[15]

[16]
[17]
[18]

[19]
[20]
[21]

A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in
neural information processing systems, 2012, pp. 1097–1105.
J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition, 2015, pp.
3431–3440.
R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,”
in Proceedings of the IEEE conference on computer vision and pattern
recognition, 2014, pp. 580–587.
A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen,
R. Prenger, S. Satheesh, S. Sengupta, A. Coates et al., “Deep
speech: Scaling up end-to-end speech recognition,” arXiv preprint
arXiv:1412.5567, 2014.
G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi,
M. Ghafoorian, J. A. van der Laak, B. van Ginneken, and C. I.
Sánchez, “A survey on deep learning in medical image analysis,”
arXiv preprint arXiv:1702.05747, 2017.
S.-A. Rebuffi, H. Bilen, and A. Vedaldi, “Learning multiple visual
domains with residual adapters,” arXiv preprint arXiv:1705.08045,
2017.
S. Zhu, C. Li, C. C. Loy, and X. Tang, “Face alignment by coarseto-fine shape searching.” in CVPR. IEEE Computer Society, 2015,
pp. 4998–5006.
K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask R-CNN,”
arXiv preprint arXiv:1703.06870, 2017.
D. Eigen and R. Fergus, “Predicting depth, surface normals and
semantic labels with a common multi-scale convolutional architecture,” in Proceedings of the IEEE International Conference on Computer
Vision, 2015, pp. 2650–2658.
H. Bilen and A. Vedaldi, “Integrated perception with recurrent
multi-task neural networks,” in Proceedings of Advances in Neural
Information Processing Systems (NIPS), 2016.
——, “Universal representations:The missing link between faces,
text, planktons, and cat breeds,” 2017.
I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, “Cross-stitch
Networks for Multi-task Learning.” CoRR, vol. abs/1604.03539,
2016.
R. M. French, “Catastrophic forgetting in connectionist networks,”
Trends in cognitive sciences, vol. 3, no. 4, pp. 128–135, 1999.
J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and
T. Darrell, “DeCAF: A Deep Convolutional Activation Feature for
Generic Visual Recognition.” in Icml, vol. 32, 2014, pp. 647–655.
A. Sharif Razavian, H. Azizpour, J. Sullivan, and S. Carlsson,
“CNN features off-the-shelf: an astounding baseline for recognition,” in Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition Workshops, 2014, pp. 806–813.
Z. Li and D. Hoiem, “Learning without Forgetting.” CoRR, vol.
abs/1606.09282, 2016.
A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive
neural networks,” arXiv preprint arXiv:1606.04671, 2016.
J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins,
A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska
et al., “Overcoming catastrophic forgetting in neural networks,”
Proceedings of the National Academy of Sciences, p. 201611835, 2017.
S. S. Sarwar, A. Ankit, and K. Roy, “Incremental Learning in Deep
Convolutional Neural Networks Using Partial Network Sharing,”
2017.
S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing
deep neural networks with pruning, trained quantization and
huffman coding,” arXiv preprint arXiv:1510.00149, 2015.
S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and
connections for efficient neural network,” in Advances in Neural
Information Processing Systems, 2015, pp. 1135–1143.

notifications,” US Patent US 9 196 164B1, 11 24, 2015.
[33] E. CYingzi Du, K. Yang, F. Jiang, P. Jiang, R. Tian, M. Luzetski,
Y. Chen, R. Sherony, and H. Takahashi, “Pedestrian behavior analysis
using 110-car naturalistic driving data in usa,” Online, 2017-063. [Online]. Available: https://www-nrd.nhtsa.dot.gov/pdf/Esv/esv23/
23ESV-000291.pdf
[22] E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus,
[34] S.
Schmidt andlinear
B. Färber,
“Pedestrians
at the
kerb–recognising
the action
“Exploiting
structure
within
convolutional
networks
for
intentions
of humans,” Transportation
F: traffic psychology
efficient evaluation,”
in Advances research
in Neuralpart
Information
Processing
and
behaviour,
vol.
12,
no.
4,
pp.
300–310,
2009.
Systems, 2014, pp. 1269–1277.
[35]
Ishaque and
and R.
“Behavioural biases
issues in
[23] M.
S. J.M.Hanson
L. B.
Y. Noland,
Pratt, “Comparing
forpedestrian
minimal
speed
choice
and streetwith
crossing
behaviour: a review,”
TransportofRenetwork
construction
back-propagation,”
in Proceedings
the
views,
vol. 28, no.Conference
1, pp. 61–85,
2008. Information Processing Systems.
1st International
on Neural
[36] T.
Wang,
J.
Wu,
P.
Zheng,
and
M.
McDonald,
“Study
of
pedestrians’
gap
MIT Press, 1988, pp. 177–185.
behaviorS.when
in
[24] acceptance
K. He, X. Zhang,
Ren,they
andjaywalk
J. Sun, outside
“Deep crossing
residual facilities,”
learning for
Intelligent
Transportation
Systems (ITSC).
IEEE,conference
2010, pp. 1295–1300.
image recognition,”
in Proceedings
of the IEEE
on computer
[37] D.
Sun,and
S. pattern
Ukkusuri,
R. F. Benekohal,
and770–778.
S. T. Waller, “Modeling of
vision
recognition,
2016, pp.
interaction
at uncontrolled
crosswalks,”
[25] motorist-pedestrian
K. Simonyan and
A. Zisserman,
“Verymid-block
deep convolutional
Urbana,
vol.for
51, p.
61801, 2002.
networks
large-scale
image recognition,” arXiv preprint
[38] E.
Papadimitriou, 2014.
G. Yannis, and J. Golias, “A critical assessment of
arXiv:1409.1556,
models,”
Transportation
research
part optimizaF: traffic
[26] pedestrian
D. Kingmabehaviour
and J. Ba,
“Adam:
A method for
stochastic
psychology
behaviour,
vol. 12, no. 3,
pp. 242–255, 2009.
tion,” arXivand
preprint
arXiv:1412.6980,
2014.
[39]
Oudejans,
C. F. Michaels,
B. van Dort,
and E. J. Frissen,
cross
[27] R.
G. R.
Griffin,
A. Holub,
and P. Perona,
“Caltech-256
object “To
category
or
not to cross:
dataset,”
2007. The effect of locomotion on street-crossing behavior,”
psychology,
8, no. 3,
pp. 259–267,
1996. layers of fea[28] Ecological
A. Krizhevsky
and G.vol.
Hinton,
“Learning
multiple
[40] A.
Lindgren,
F. Chen,
P. W.2009.
Jordan, and H. Zhang, “Requirements for
tures
from tiny
images,”
design ofand
advanced
assistance
systems-the differences
[29] the
S. Munder
D. M.driver
Gavrila,
“An experimental
study onbetween
pedesSwedish
and Chinese drivers,”
International
Design,and
vol.ma2,
trian classification,”
IEEE transactions
onJournal
pattern ofanalysis
no.
2, intelligence,
2008.
chine
vol. 28, no. 11, pp. 1863–1868, 2006.
[41]
M. Trivedi, T.M.
B. Schlipsing,
Moeslund et al.,
“Trajectory
analysis
and prediction
[30] M.
J. Stallkamp,
J. Salmen,
and
C. Igel,
“Man vs.
for
improvedBenchmarking
pedestrian safety:
Integrated
framework
and evaluations,”
computer:
machine
learning
algorithms
for traffic
in
Intelligent
Vehicles
Symposium
(IV).vol.IEEE,
2015,
pp. 330–335.
sign
recognition,”
Neural
networks,
32, pp.
323–332,
2012.
[42]
Kooij,K.N.Kavukcuoglu,
Schneider, andD.D.Silver,
M. Gavrila,
“Analysis
of pedestrian
[31] J.
V. F.Mnih,
A. A. Rusu,
J. Veness,
M. G.
dynamics
from
vehicle M.
perspective,”
in A.
Intelligent
VehiclesG.Symposium
Bellemare,
A. aGraves,
Riedmiller,
K. Fidjeland,
Ostrovski
(IV).
IEEE, 2014, pp.control
1445–1450.
et al., “Human-level
through deep reinforcement learning,”
[43] B.
Völz,vol.
K. Behrendt,
H. Mielenz,
I. Gilitschenski,
R. Siegwart, and
Nature,
518, no. 7540,
pp. 529–533,
2015.
“A data-driven
approach
for pedestrian
estimation,”
[32] J.
R.Nieto,
K. Cowen,
S. Sponaugle,
K. Robinson,
and intention
J. Luo, “Planktonset
1.0: Plankton imagery data collected from fg walton smith in
straits of florida from 2014–06-03 to 2014–06-06 and used in the
2015 national data science bowl (ncei accession 0127422),” Oregon
State University; Hatfield Marine Science Center, 2015.
[33] M. Eitz, J. Hays, and M. Alexa, “How do humans sketch objects?”
ACM Trans. Graph., vol. 31, no. 4, pp. 44–1, 2012.
[34] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y.
Ng, “Reading digits in natural images with unsupervised feature
learning,” in NIPS workshop on deep learning and unsupervised feature
learning, vol. 2011, no. 2, 2011, p. 5.
[35] A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht,
“The Marginal Value of Adaptive Gradient Methods in Machine
Learning.” CoRR, vol. abs/1705.08292, 2017.
[36] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “Imagenet
large scale visual recognition challenge,” International Journal of
Computer Vision, vol. 115, no. 3, pp. 211–252, 2015.
[37] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi,
“Fine-grained visual classification of aircraft,” arXiv preprint
arXiv:1306.5151, 2013.
[38] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi,
“Describing Textures in the Wild.” CoRR, vol. abs/1311.3618, 2013.
[39] M.-E. Nilsback and A. Zisserman, “Automated Flower Classification over a Large Number of Classes.” in ICVGIP. IEEE, 2008, pp.
722–729.
[40] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum, “Humanlevel concept learning through probabilistic program induction,”
Science, vol. 350, no. 6266, pp. 1332–1338, 2015.
[41] K. Soomro, A. R. Zamir, and M. Shah, “UCF101: A dataset of 101
human actions classes from videos in the wild,” 2012.
[42] S. Zagoruyko and N. Komodakis, “Wide Residual Networks.”
CoRR, vol. abs/1605.07146, 2016.
[43] J. Zhang, I. Mitliagkas, and C. Ré, “YellowFin and the Art of
Momentum Tuning,” arXiv preprint arXiv:1706.03471, 2017.

Amir Rosenfeld received his B.Sc
in Computer-Engineering at the Hebrew
University of Jerusalem (HUJI) at 2006
and his M.Sc. in Computer Science at
HUJI in 2011. He completed his PhD
in Computer Science at the Weizmann
Institute under the supervision of Prof.
Shimon Ullman in 2016. He is currently
a postdoctoral fellow at the Laboratory for Active and Attentive Vision, York University. His main research interests

and Attentive Vision, York University. Her research
interests include visual attention, computer vision,
cognitive systems and autonomous driving.
13

include machine learning, computer and human vision,
visual attention and adaptive learning systems.
John K. Tsotsos is Distinguished Re-

John
K. Tsotsos
is Distinguished
Research
Profes-at
search
Professor
of Vision
Science
sor of Vision Science at York University. He received
York
University.
He
received
his
doctorhis doctorate in Computer Science from the UniverateofinToronto.
Computer
from
the Unisity
After aScience
postdoctoral
fellowship
in
Cardiology
at Toronto.
Toronto General
he joined
versity of
AfterHospital,
a postdoctoral
the University of Toronto on faculty in Computer
fellowship in Cardiology at Toronto
Science and in Medicine. In 1980 he founded the
GeneralVision
Hospital,
theof UniverComputer
Group athe
thejoined
University
Toronto,
which
he Toronto
led for 20on
years.
He was
to
sity of
faculty
in recruited
Computer
York
University
2000
as Director In
of the
Centre
Science
andin in
Medicine.
1980
he
for Vision Research. He has been a Canadian Heart
founded
the
Vision
Foundation Research Scholar,
Fellow of
theComputer
Canadian Institute
for Group
Advancedat
the University
Toronto,
he led for
20 He
years.
He
Research
and Canadaof
Research
Chairwhich
in Computational
Vision.
received
many
and honours
including
several best
paper awards,
the 2006of
was awards
recruited
to York
University
in 2000
as Director
Canadian Image Processing and Pattern Recognition Society Award for
the Centre for Vision Research. He has been a Canadian
Research Excellence and Service, the 1st President’s Research Excellence
Heartby Foundation
Scholar,
Fellow
of the
CanaAward
York University Research
in 2009, and the
2011 Geoffrey
J. Burton
Memorial
Lectureship
from for
the United
Kingdom’s
Applied
Vision
Association
for
dian Institute
Advanced
Research
and
Canada
Research
significant
to vision Vision.
science. He
elected as
Fellowawards
of the
Chair incontribution
Computational
Hewas
received
many
Royal Society of Canada in 2010 and was awarded its 2015 Sir John William
and honours
severalinbest
paper awards,
Dawson
Medal forincluding
sustained excellence
multidisciplinary
research,the
the 2006
first
Canadian
Image
Recognition
computer
scientist
to beProcessing
so honoured.and
OverPattern
125 trainees
have passed Society
through
his
lab. Hisfor
current
researchExcellence
focuses on a and
comprehensive
of visual
Award
Research
Service, theory
the 1st
Presattention in humans. A practical outlet for this theory embodies elements of
idents
Excellence
Award
by York University in
the
theory Research
into the vision
systems of mobile
robots.

2009, and the 2011 Geoffrey J. Burton Memorial Lectureship
from the United Kingdoms Applied Vision Association for
significant contribution to vision science. He was elected
as Fellow of the Royal Society of Canada in 2010 and
was awarded its 2015 Sir John William Dawson Medal for
sustained excellence in multidisciplinary research, the first
computer scientist to be so honoured. Over 125 trainees
have passed through his lab. His current research focuses
on a comprehensive theory of visual attention in humans.
A practical outlet for this theory embodies elements of the
theory into the vision systems of mobile robots.

