Investigating the Impact of Data Volume and Domain
Similarity on Transfer Learning Applications
Michael Bernico∗, Yuntao Li†, Dingchao Zhang‡

arXiv:1712.04008v2 [cs.CV] 3 Jan 2018

State Farm Mutual Insurance

Abstract
Transfer learning helps to build a system to recognize and apply knowledge and experience learned in previous tasks (source task) to new tasks or new domains (target task),
which share some commonality. The two important factors that impact the performance
of transfer learning models are: (a) the size of the target dataset and (b) the similarity
in distribution between source and target domains. Thus far there has been little investigation into just how important these factors are. In this paper, we investigated the
impact of target dataset size and source/target domain similarity on model performance
through a series of experiments. We found that more data is always beneficial, and that
model performance improved linearly with the log of data size, until we were out of data.
As source/target domains differ, more data is required and fine tuning will render better
performance than feature extraction. When source/target domains are similar and data
size is small, fine tuning and feature extraction renders equivalent performance. We hope
that our study inspires further work in transfer learning, which continues to be a very
important technique for developing practical machine learning applications in business
domains.

1

Introduction

For many applications of computer vision, data is in short supply. While Convolutional
Neural Networks (ConvNets) trained on large datasets such as ImageNet [1] with the benefit of
massive computational power have revolutionized computer vision, many business applications
of computer vision has limited relevant data. Many researchers and machine learning engineers
have had great success in addressing this problem by utilizing transfer learning [2].
With transfer learning the knowledge from a network trained on a large dataset such as
ImageNet (source domain) are transferred to another problem domain (target domain). There
are a variety of techniques that can be applied to accomplish this, however most commonly
the final network layers of the original network are replaced with layers more suitable for the
target domain and the network is then trained on data related to that target domain. This
network benefits from the learnings of the original network, allowing the network to perform
its task using significantly less training data.
∗

michael.bernico.qepz@statefarm.com
yuntao.li.e0b5@statefarm.com
‡
dingchao.zhang.ejlq@statefarm.com
†

1

1.1

Related Work

This study is motivated nearly entirely by the work done by [3], in “Revisiting Unreasonable
Effectiveness of Data in Deep Learning Era.” In this paper, the authors noted that while model
size and computational power have increased over the previous 5 years, ImageNet is still being
used to train those models. The paper goes on to examine the impact of increasing the volume
of training data from ImageNet’s 10 million images to 300 million images in the Google JFT300M image dataset. In doing so, the authors observed that model performance increases
logarithmically based on the volume of training data. These results sparked our curiosity. We
wanted to understand the impact that target domain data volume and source/target domain
similarity had on model performance.

2

Experiments

To understand the impact of data volume, and relationships between target and source data
domains in transfer learning problems, various pre-trained neural network architectures are
selected and further tuned with new training data of different domains and sizes over a series
of experiments. Model performances are subsequently compared and analyzed to obtain more
insights.

2.1

Datasets

Two datasets are primarily used throughout this study, namely the MiniPlace dataset and
IMDB-WIKI dataset, although internally enhanced and preprocessed. The Kaggle’s Dogs vs.
Cats (Dogs/Cats) data are also used as an exploratory dataset.
The Dogs/Cats dataset is from Kaggle’s Dogs vs. Cats competition, originally provided
by Microsoft Research[4]. The dataset consists of 25000 training images of cats and dogs,
creating a binary classification problem. A separate unlabeled test set was provided but not
used.
The MiniPlace data set originates from the MIT Places2 data set[7], a scene-centric
database with more than 10 million images comprising 400+ unique scene categories. MiniPlace is a scaled-down version of Places2, with 100,000 images of 100 categories with a resolution of 128 × 128.
The IMDB-WIKI dataset is collected and published by Rothe et al[5], which is by far
the largest publicly available dataset of face images with gender and age labels for training.
The dataset consists of public celebrity facial images with age and gender labels inferred
from associated timestamps and names, and totals at 523,051 images. As some of the images
contain multiple faces or suffer from low quality issues, the entire dataset is internally cleaned
and preprocessed, as well as merged with internal employee profile image data to enhance its
size (∼ 1 million images) and quality. In this study, only age labels are used.
As one of the most common issues encountered in business applications is the lack of data,
where a typical data set would be on the order of 10k to 100k records, only a subset (100k)
of both datasets are used throughout the experiments to investigate the effects of data on
transfer learning in a similar data volume regime.

2

2.2

Neural Network Models

Different Neural Network architectures are tested in this study. Google’s InceptionV3 model[8]
pre-trained on ImageNet (2012) is tested on the Dogs/Cats dataset as an exploratory analysis
on impact of data size. The VGG-Face model[6] pre-trained on public internet data is applied
to the IMDB-WIKI dataset as an example of applications with highly similar target and
source domains. In addition, the VGG-Face model is also applied to the MiniPlace data
to investigate transfer learning applications with highly different target and source domains.
The VGG16 model[9] pre-trained using ImageNet data is applied to the MiniPlace data as an
example use case with moderately different target and source domains.

2.3

Training Process

A series of transfer learning experiments with different target/source domains and varied
training data sizes are conducted to shed lights on the effect of data size on transfer learning
performances. The experiments can be categorized into three groups depending on how
different the target domain is form the source domain.
Throughout this paper, two transfer learning strategies are utilized and studied, as defined
below:
Feature extraction: Using a ConvNet pre-trained on source domain data, we remove the
last fully connected layer and output layer. Those layers are then replaced with a new fully
connected layer and target domain specific output layer. The remaining layers are used as a
fixed feature extractor, while only the replaced layers are trained.
Fine-tuning: After accomplishing the training steps specified in feature extraction, we
then allow the pretrained layers of the ConvNet to be updated via back-propagation, fine
tuning their weights for the target domain. It is possible to fine-tune all the layers of the
ConvNet, or to keep some of the earlier layers fixed and only fine-tune some higher-level
portion of the network.
For highly similar source/target domains, the VGG-Face model is trained on the IMDBWIKI dataset with increasing data size from 10k to 100k incremented by 10k, to predict the
individuals’ ages. Fine-tuning and feature extraction are respectively applied to each of the
experiments.
For highly different source/target domains, the VGG-Face model is trained on the MiniPlace dataset with increasing data size from 10k to 100k incremented by 10k (random sampling), to classify the different scenes contained in the data. Fine-tuning and feature extraction
are respectively applied to each of the experiments.
For moderately different source/target domains, the VGG16 model is trained on the MiniPlace dataset with increasing data size from 10k to 100k incremented by 10k, to classify the
different scenes contained in the data. Fine-tuning and feature extraction are respectively
applied to each of the experiments.
All training processes utilized the “ADAM”[10] optimizer with number of epochs set to
10 to ensure consistent comparisons across all models. Model performances are evaluated
on a holdout dataset of 10k in size. The ‘categorical accuracy’ metric is used for evaluating
classification models, while the ‘mean absolute error’ metric is used to evaluating regression
models.

3

3

Results and Discussions

3.1

Impact of Data Size

For this experiment, all model parameters are fixed except for the size of the training data,
which grows from 10k to 100k with an increment step of 10k. The experiment is conducted on
VGG-Face, VGG16, and Inception V3 models with IMDB-WIKI, MiniPlaces , and Dogs/Cats
datasets respectively. Figure 1 shows the improvement of model accuracy as the training data
size grows. Not surprisingly, with the same number of epochs and learning rate, the model
accuracy continues increasing as training data expands, which demonstrates the effectiveness
of data size. In addition, the lift in model performance scales logarithmically with the data
size, in consistency with the findings in [3].

(a) Source: VGG-Face; Target: IMDB-WIKI

(b) Source: VGG16; Target: MiniPlace

(c) Source: Inception V3; Target: Dogs/Cats

Figure 1: Model performance is logarithmically correlated to training data size. The VGGFace model is shown on the top left, while the VGG16 model is shown on the top right. The
lower left shows the results from the InceptionV3 model

3.2

Highly Similar Target and Source Domains

For this experiment, fine-tuning and feature extraction are each applied to training the VGGFace model with the IMDB-WIKI dataset, to predict individuals’ ages in the images. The
4

models are trained with 10 epochs with expanding training data from 10k to 100k. Since the
original data used in VGG-Face model and the IMDB-WIKI dataset both consists of facial
images, this experiment serves as an investigation into transfer learning applications with
highly similar target/source domains.
Figure 2 shows the decay of loss function over the training epochs for each experiment,
where results from feature extraction are illustrated in the left two subfigures, and fine-tuning
is shown to the right. It can be seen that both the training and validation loss values using
feature extraction tend to stabilize and plateau by the end of 10 epochs, as more training data
become available. Seemingly, the transition point is roughly when training data contains 80k
images. Contrarily, fine-tuning leads to an ever-decreasing train and validation loss indicating
that the model performance can be further improved if more data and training time are
provided. Such observations indicate that for transfer learning problems with highly similar
target and source domains, it would be beneficial to perform fine-tuning rather than feature
extraction when the training data becomes large enough. For this particular combination
of VGG-Face model and IMDB-WIKI dataset, the transition training size is around 80k,
however, it is possible that a corresponding training size exists for every transfer learning
setting, above which fine-tuning gives better model performance while feature extraction can
still produce similar results when the training data are below the transition point.

5

(a) Feature extraction

(b) Fine-tuning

Figure 2: Source: VGG-Face; Target: IMDB-WIKI. Comparison of feature extraction and
fine-tuning in reduction of loss function values. The decreasing rates of both training and
validation loss diminish as training data grows under feature extraction, while the losses keep
declining in fine-tuning.

3.3

Moderately Different Target and Source Domains

For this experiment, fine-tuning and feature extraction are each applied to training the VGG16
model with the MiniPlace data, to classify image scenes in the data. The models are trained
with 10 epochs with expanding training data from 10k to 90k. ImageNet is the source data
VGG16 is trained on, which contains thousands of image categories including some similar
scenes in the MiniPlace data, while other categories differ significantly from the MiniPlace images. Therefore, this experiment serves as an investigation into transfer learning applications
with target/source domains sharing limited similarities, but still being largely different.
The experimental results are shown in Figure 3 with feature extraction plotted on the left
and fine-tuning on the right. These plots compare the evolution of both training validation
with respect to training epochs under both feature extraction and fine-tuning. It can be
seen that there is a sizable gap between training and validation loss, which are also large in
absolute values in both plots. This implies the model suffers from higher bias compared to
the last experiment, meaning that training becomes more difficult as target and source data
domains diverge. Furthermore, the final loss values in both training and validation are lower

6

in the case of fine-tuning than feature extraction, although the difference is not so obvious.
It is observed that fine-tuning rendered similar training and validation losses compared to
feature extraction, when the training size is small; when training size gets more than 60,000,
fine-tuning starts out-perform feature extraction with lower validation losses generated. Additionally, overfitting is present in both experiments, where training loss increases as the size
of training data increases monotonically, which can be explained by the small size of training data, noted that we deliberately compared the performance of fine-tuning and feature
extraction across different sizes from small to large in experiments.

Figure 3: Source: VGG16; Target: MiniPlace. Comparison of feature extraction and finetuning in reduction of loss function values. The left panel shows the results of feature extraction, while the right panel shows the results of fine-tuning. Both training and validation loss
are slightly lower in fine-tuning than feature extraction by the end of training.

3.4

Highly Different Target and Source Domains

For this experiment, fine-tuning and feature extraction are each applied to training the VGGFace model with the MiniPlace dataset, to classify image scenes in the data. The models are
trained with 10 epochs with expanding training data from 10k to 90k. As aforementioned,
the source data used in VGG-Face model primarily contains facial information, distinctively
different than the MiniPlace data which contains general scenes such as airport and county
yard, this experiment is hence an investigation in to transfer learning applications with target/source domains dramatically different from each other.
Results from both fine-tuning and feature extraction are summarized in Figure 4, showing
the decay of validation loss function over the training epochs for each experiment. Feature
extraction is plotted on the left, whereas fine-tuning is shown on the right. Comparing to
the last experiment, it becomes more difficult to achieve similar accuracies with the same
number of epochs in training. As the target/source domains become more different, the level
of difficulty in training increases.
Although the validation loss increases over training epochs (an implication of overfitting)
and no obvious differences in model performance are observed between feature extraction and
fine-tuning, there seems to be a critical training size of 60k approximately, where validation
loss exhibits a dip starting between training epoch 3 and 4. In addition, as the training size
becomes larger, the increase in validation loss slows down and eventually starts decreasing
when training size approaches 80k or 90k. Such observations indicate that early stopping
7

may be beneficial in this problem setting when data size is limited. Having larger training
data is the key to better model performances and 100k is still not large enough to uncover
performance differences between feature extraction and fine-tuning.

(a) feature extraction

(b) Fine-tuning

Figure 4: Source: VGG-Face; Target: MiniPlace. Comparison of feature extraction and finetuning in reduction of loss function values. A dip in validation loss starting around training
epoch 3 is observed when training size is larger than 60k in both sets of experiments.

4

Conclusions

In this study we have investigated the impact of target data size in a typical business application of transfer learning where training data are very much limited. In addition, we also
studied how similarity between target and source data domains could affect the final model
performance by using different transfer learning training approaches, namely feature extraction and fine-tuning. From a series of experiments, we find that in general, a larger target
data size will always lead to better training convergence, although the marginal performance
decreases as the dataset grows. More precisely, there is an apparent logarithmic relationship
between model performance and the target data sizes. The level of similarity between the
source and target data domains also plays a vital role in selecting the appropriate training
framework. When target and source domains are similar, feature extraction gives equivalent
training results as fine-tuning in small training data settings. However, fine-tuning tends to
8

give better results as data size increases and it is uncovered that each specific target/source
combination likely has a unique such transition data size. When target and source domains
diverge, it’s crucial to collect more training data. While both feature extraction and finetuning are comparable throughout all experiments, the validation performance would only
increase when target training data are sufficiently large.
Data is a precious asset, but unfortunately it is not always sufficiently available or at its
best quality, especially in most real-world business problems. Although more detailed research
is surely essential to fully uncover the quantitative impact of target data size and differences
between feature extraction and fine-tuning, our study still provides some insights into how to
appropriately apply transfer learning techniques in typical business application settings and
also motivate more and more business to explore the significance of data in developing more
accurate statistical models.

References
[1] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy,
A. Khosla, M. S. Bernstein, A. C. Berg, and F. Li. Imagenet large scale visual recognition
challenge. arXiv:1409.0575, 2014.
[2] S. J. Pan, Q. Yang. A survey on transfer learning. IEEE Transactions on knowledge and
data engineering, 22(10):1345 –1359, 2010.
[3] Chen Sun, Abhinav Shrivastava, Saurabh Singh, Abhinav Gupta. Revisiting Unreasonable
Effectiveness of Data in Deep Learning Era. ICCV, 2017
[4] Jeremy Elson, John R. Douceur, Jon Howell, Jared Saul, Asirra. A CAPTCHA that Exploits Interest-Aligned Manual Image Categorization. Proceedings of 14th ACM Conference
on Computer and Communications Security (CCS), Association for Computing Machinery,
Inc., 2007
[5] Rasmus Rothe and Radu Timofte and Luc Van Gool. Deep expectation of real and apparent age from a single image without facial landmarks. International Journal of Computer
Vision (IJCV), 2016.
[6] O. M. Parkhi, A. Vedaldi, A. Zisserman. Deep Face Recognition. British Machine Vision
Conference, 2015.
[7] Zhou, Bolei and Lapedriza, Agata and Khosla, Aditya and Oliva, Aude and Torralba,
Antonio. Places: A 10 million Image Database for Scene Recognition. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 2017.
[8] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, Zbigniew Wojna.
Rethinking the Inception Architecture for Computer Vision. arXiv:1512.00567, 2015
[9] K. Simonyan, A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image
Recognition. ICLR, 2015
[10] Diederik P. Kingma, Jimmy Ba. Adam:
arXiv:1412.6980, 2014

9

A Method for Stochastic Optimization.

