RankIQA: Learning from Rankings for No-reference Image Quality Assessment
Xialei Liu Joost van de Weijer Andrew D. Bagdanov
cs.CV







RankIQA: Learning from Rankings for No-reference Image Quality Assessment
    Xialei Liu

Computer Vision Center 

Barcelona, Spain

xialei@cvc.uab.es





Joost van de Weijer

Computer Vision Center

Barcelona, Spain

joost@cvc.uab.es
Andrew D. Bagdanov


MICC, University of Florence 

Florence, Italy

andrew.bagdanov@unifi.it

    
================================================================================================================================================================================================================================================================






 We propose a no-reference image quality assessment
  (NR-IQA) approach that learns from rankings 
  (RankIQA). To address the problem of limited IQA dataset size, we
  train a Siamese Network to rank images in terms of image quality by
  using synthetically generated distortions for which relative image
  quality is known. These ranked image sets can be automatically
  generated without laborious human labeling. We then use
  fine-tuning to transfer the knowledge represented in the trained
  Siamese Network to a traditional CNN that estimates absolute image
  quality from single images. We demonstrate how our approach can be
  made significantly more efficient than traditional Siamese Networks
  by forward propagating a batch of images through a single network
  and backpropagating gradients derived from all pairs of images in
  the batch. Experiments on the TID2013 benchmark show that we improve the state-of-the-art by over 5%. Furthermore, on the LIVE benchmark we show that our approach is superior to existing NR-IQA techniques and that we even outperform the state-of-the-art in full-reference IQA (FR-IQA) methods without having to resort to high-quality reference images to infer IQA.





§ INTRODUCTION


Images are everywhere in our life. Unfortunately, they are often
distorted by the processes of acquisition, transmission, storage, and
external conditions like camera motion. Image Quality
Assessment (IQA) <cit.> is a technique developed to
automatically predict the perceptual quality of images. IQA estimates
should be highly correlated with quality assessments made by a range
of very many human evaluators (commonly referred to as the Mean
Opinion Score
(MOS) <cit.>). IQA has
been widely applied to problems where image quality is essential, like image
 restoration <cit.>, image
super-resolution <cit.>, and image
retrieval <cit.>.



IQA approaches are generally divided into three categories based on
whether the undistorted image (called reference image) or information about it is available:
full-reference IQA (FR-IQA), reduced-reference IQA (RR-IQA), and
no-reference IQA (NR-IQA). Research has mostly focussed on the more realist scenario of NR-IQA where the image
quality of an image without any reference image has to be estimated.
In NR-IQA, many methods focus on a specific
distortion <cit.>, which limits the
applicability of these methods. Other methods consider a range
of distortions <cit.>.



Convolutional Neural Networks (CNNs) are having an enormous impact on
computer vision research and practice. Though they have been around
for decades <cit.>, it wasn't until 2012, when
Krizhevsky et al. <cit.> achieved spectacular
results with a CNN in the ImageNet competition, that they achieved
wide attention and adoption in the broader computer vision
community. The architectures of networks are getting deeper and deeper
with respect to the original AlexNet, with ResNet being an example of
very deep network architecture <cit.>. The result of this
trend is that state-of-the-art CNNs like AlexNet and ResNet have
hundred of millions of parameters and require massive amounts of data
to train from scratch (without overfitting).




The success of CNNs encouraged research exploring their potential
application to the NR-IQA problem. This research resulted in significant
improvements compared to previous hand-crafted approaches  <cit.>.
The main problems these papers had to address is the absence of large datasets for IQA .
Especially as networks grow deeper and wider, the number of parameters increases dramatically. As a
consequence, larger and larger annotated datasets are required for
training. However, the annotation process for IQA image datasets
requires multiple human annotations for every image, and thus the
collection process is extremely labor-intensive and costly. As a
results, most available IQA datasets are too small
to be effective for training CNNs.



We propose an approach to address the absence of large datasets.
The main idea (see Fig. <ref>) is that while human-annotated IQA data is difficult to
obtain, it is easy to generate images that are
ranked according to their image quality. That is, we can
generate image sets in which, though we do not have an absolute
quality measure for each generated image, for any pair of
images we know which is of higher quality. We call this
learning from rankings approach RankIQA, and with it we learn to rank image in terms of quality using Siamese Networks, and then we transfer knowledge
learned from ranked images to a traditional CNN fine-tuned on IQA data in order to improve the accuracy of IQA. The idea to learn IQA features from distorted reference images was proposed by Zhang et al. in a patent <cit.>. In this paper we go beyond this patent in that we provide a detailed description of our method and experimentally verify the usefulness of pre-training networks using ranked datasets.

As a second contribution we
propose a method for efficient backpropagation in Siamese networks. The method
forwards a batch of images through a single network and then backpropagates gradients derived from
all pairs in the batch.
In extensive experiments on established IQA datasets
we show that learning from rankings significantly improves results, and that our efficient backpropagation
algorithm allows to train these networks better and faster than other training protocols, like hard-negative mining. The supplementary material and project page are available at urlhttps://github.com/xialeiliu/RankIQA.



§ RELATED WORK

We briefly review the literature related to our approach. We focus on
distortion-generic NR-IQA since it is more generally applicable than
the other IQA research lines.


Traditional NR-IQA approaches.  Most traditional NR-IQA
can be classified into Natural Scene Statistics (NSS) methods and
learning-based methods. In NSS methods, the assumption is that images
of different quality vary in the statistics of responses to specific
filters. Wavelets <cit.>, DCT <cit.> and
Curvelets <cit.> are commonly used to extract the features
in different sub-bands. These feature distributions are parametrized,
for example with the Generalized Gaussian
Distribution <cit.>. The aim of these methods is
to estimate the distributional parameters, from which a quality
assessment can be inferred. The authors of <cit.> propose to extract NSS features in the spatial domain to obtain
significant speed-ups. In learning-based methods, local features are extracted and mapped to
the MOS using, for example, Support Machine Regression
or Neural Networks <cit.>. The codebook
method <cit.> combines different features
instead of using local features directly. Datasets without MOS can be
exploited to construct the codebook by means of unsupervised learning,
which is particularly important due to of the small size of existing
datasets. Saliency maps <cit.> can be used to model human
vision system and improve precision in these methods.






























Deep learning for NR-IQA.  In recent years several works
have used deep learning for
NR-IQA <cit.>.
One of the main drawbacks of deep networks is the need for large
labeled datasets, which are currently not available for NR-IQA
research. To address this problem Kang et
al. <cit.> consider small 32 × 32 patches
rather than images, thereby greatly augmenting the number of training
examples. The authors of <cit.>
follow the same pipeline. In <cit.> the authors
design a multi-task CNN to learn the type of distortions and image
quality simultaneously. Bianco at al. <cit.> propose to
use a pre-trained network to mitigate the lack of training data. They
extract features from a pre-trained model fine-tuned on an IQA
dataset. These features are then used to train an SVR model to map
features to IQA scores. 

In our paper, we propose a radically different
approach to address the lack of training data: we use a large number
of automatically generated rankings of image quality to train a deep
network. This allows us to train much deeper and wider networks than
other methods in NR-IQA which train directly on absolute IQA data.










Learning to rank. These approaches learn a ranking
function from ground-truth rankings by minimizing a ranking
loss <cit.>. This function can then be applied to rank
test objects. The authors of <cit.> adapt the
Stochastic Gradient Descent method to perform pairwise learning
to rank. This has been successfully applied to large
datasets. Combining ideas from ranking and CNNs, the Siamese network
architecture achieves great success on the face verification
problem <cit.>, and in comparing image
patches <cit.>. The only other work which
applies rankings in the context of NR-IQA is <cit.> in which they combine different hand-crafted
features to represent image pairs from the IQA dataset.

Our approach is different in that primarily we are not aiming to learn
rankings. Instead we use learning from rankings as a data augmentation
technique: we use easily obtainable datasets of ranked images to
train a large network, which is then fine-tuned for the task of
NR-IQA.





Hard-negative mining for Siamese network training. It is
well known that a naive approach to sampling pairs to training Siamese
networks is suboptimal. To address this problem several approaches to
hard-negative mining have been
proposed. In <cit.>, they propose a hard
positive and hard-negative mining strategy to forward-propagate a set
of pairs and sample the highest loss pairs with back-propagation.
However, hard mining comes with a high computational cost (they report
an increase of up to 80% of total computation
cost). In <cit.> they propose semi-hard pair
selection, arguing that selecting hardest pairs can lead to bad local
minima. The batch size used is around 1800 examples, which again leads
to a considerable increase in computational
cost. In <cit.> the authors take a batch of
pairs as input and choose the four hardest negative samples within the
mini-batch. To solve for a bad local optimum, <cit.>
optimize a smooth upper bound loss function to take advantage of all
possible pairs in the mini-batch.

In contrast with these works, we propose a method for efficient
Siamese backpropagation which does not depend on hard-negative
selection. Instead, it considers all possible pairs in the
mini-batch. This has the advantage that the main computational
bottleneck in training deep networks, namely the forward-propagation
of images through the network, is optimally exploited.




§ LEARNING FROM RANKINGS FOR NR-IQA



In this section we describe our approach to exploiting synthetically
generated rankings for NR-IQA. We first lay out a general framework
for our approach, then describe how we use a Siamese network
architecture to learn from rankings. Finally, in
section <ref> we show how backpropagation for
training Siamese networks from ranked samples can be made
significantly more efficient.



 §.§ Overview of our approach

The lack of large IQA datasets motivates us to propose a new strategy
to take advantage of large, unlabelled databases from which we
can generate images ranked by image quality. Our approach is based on
the observation that, given a set of arbitrary reference images, it is
very easy to apply image distortions to generate a ranking
image dataset. As an example, given a reference image we can apply
various levels of Gaussian blur. The set of images which is thus
generated can be easily ranked because we do know that adding Gaussian
blur (or any other distortion) always deteriorates the quality
score. Note that in such set of ranked images we do not have any
absolute IQA scores for any images – but we do know for any pair of
images which is of higher quality.

After learning on these ranked images, we can use fine-tuning on small
image quality datasets in order to address the IQA problem. The
difference between our approach and the straightforward, classical
approach <cit.> is shown in
Fig. <ref>. The standard approach trains a shallow network
directly on the IQA dataset to estimate IQA score from images. Due to
the limited data only few layers can be used, which limits
accuracy. Since we have access to much larger datasets with ranked
images, we can now train deeper and wider networks to learn a distance
embedding. Next we follow this by fine-tuning for domain adaptation to
the absolute IQA problem. The overall pipeline of our approach is:



  * Synthesize ranked images. Using an arbitrary set of
  images, we synthetically generate deformations of these
  images over a range of distortion intensities. The absolute
  distortion amount for each image is not used in subsequent steps,
  but within each deformation type we know, for any pair of images,
  which is of higher quality. See section <ref> for a
  description of the datasets used for generating ranked images and
  the distortions applied.

  * Train Siamese network for ranking. Using the set of
  ranked images, we train a Siamese network described in the next
  section using the efficient Siamese backpropagation technique
  proposed in section <ref>. The result is a Siamese
  network that ranks images by image quality. 

  * Fine-tune on IQA data. Finally, we extract a single
  branch of the Siamese network (we are interested at this point in
  the representation learned in the network, and not in the ranking
  itself), and fine-tune it on available IQA data. This effectively
  calibrates the network to output IQA measurements.




 §.§ Siamese networks for ranking

Here we introduce the Siamese network <cit.> to learn from
image rankings, which is a
network with two identical network branches and a loss module. The two
branches share weights during training. Pairs of images and labels are
the input of the network, yielding two outputs which are passed to the
loss module. The gradients of the loss function with respect to all
model parameters are computed by backpropagation and updated with the
stochastic gradient method.




















Specifically, given an image x as the input of the network, the
output feature representation of x, denoted by f(x;θ), is
obtained by capturing the activation in the last layer. Here θ
are the network parameters, and we will use y to denote the ground truth value for the image which for image quality assessment is its quality score. Consequently, in our Siamese networks the output of the
final layer is always a single scalar which we want to be indicative
of the image quality.  Since our aim is to rank the images, we apply
the pairwise ranking hinge loss:

    L(x_1, x _ 2 ;θ) = max(0,f(x_2;θ)-f( x_ 1 ;θ)+ε )

where ε is the margin. Here we assume without loss of generality that the rank of x_1
is higher than x_2. The gradient of the loss in
Eq. <ref> is given by:


    0.8!∇_θL =
    
      0                  if  f(  x _ 2 ;θ) - f(  x
        _ 1 ;θ) + ε≤ 0 
    ∇_θf(  x _ 2 ;θ) -∇_θf(  x _ 1 ;θ)    otherwise.

In other words, when the outcome of the network is in accordance with
the ranking, the gradient is zero. When the outcome of the network is
not in accordance we decrease the gradient of the higher and add the
gradient of the lower score. Given this gradient of L with respect
to model parameters θ, we can train the Siamese network using
Stochastic Gradient Descent (SGD).













 §.§ Efficient Siamese backpropagation



One drawback of Siamese networks is the redundant
computation. Consider all possible image pairs constructed from three
images. In a standard implementation all three images are passed twice
through the network, because they each appear in two pairs. Since both
branches of the Siamese network are identical, we are essentially
doing twice the work necessary since any image need only be passed
once through the network. It is exactly this idea that we
exploit to render backpropagation more efficient. In fact, nothing prevents us from considering all possible
pairs in the mini-batch, without hardly any additional computation.
We add a new layer to the network that generates all possible pairs in
a mini-batch at the end of the network right before computing the
loss. This eliminates the problem of pair selection and boosts
efficiency.










To approximate the speed-up of efficient Siamese backpropagation
consider the following. If we have one reference image distorted by
n levels of distortions in our training set, then for a traditional
implementation of the Siamese network we would have to pass a total of
n ^ 2 -n images through the network – which is twice the
number of pairs you can generate with n images. Instead we propose
to pass all images only a single time and consider all possible pairs
only in the loss computation layer.  This reduces computation to just
n passes through the network. Therefore, the speed-up is equal to:
n ^ 2 -n / n  =  n -1. In the best scenario n=M,
where M is equal to the number of images in the mini-batch, and
hence the speed-up of this method would be in the order of the
mini-batch size. Due to the high correlation among the set of all
pairs in a mini-batch, we expect the final speedup in convergence to
be lower.












To simplify notation in the following, we let
ŷ_i = f(x_i;θ), where f(x_i;θ) is the output of a
single branch of the Siamese network on input x_i. Now, for one pair
of inputs the loss is:

    L(x_1,x_2,l_12; θ) = g(ŷ_1,ŷ_2,l_12),

where l_12 is a label indicating the relationship between image
1 and 2 (for example, for ranking it indicates whether x_1 is of
higher rank than x_2), and
θ = {θ_1, θ_2, …, θ_k} are all model
parameters in the Siamese network. We omit the θ dependency in
g for simplicity. The gradient of this loss function with respect to
the model parameter θ is:

    ∇_θ L=
      ∂ g(ŷ_1,ŷ_2,l_12) /∂ŷ_1 ∇_θŷ_1
      +
      ∂ g(ŷ_1,ŷ_2,l_12) /∂ŷ_2 ∇_θŷ_2.

























This gradient of L above is a sum since the model parameters
  are shared between both branches of the Siamese network and
  ŷ_1 and ŷ_2 are computed using exactly the same
  parameters.

Considering all pairs in a mini-batch, the loss is:

    L = ∑_i=1^M∑ _ j > i ^ M  g(ŷ_i,ŷ_j,l_ij)


The gradient of the mini-batch loss with respect to parameter θ
can then be written as:














    ∇_θ L  =
    ∑_ i=1 ^ M ∑_ j > i ^M
     ∂ g(ŷ_i,ŷ_j, l_ij) /∂ŷ_i∇_θŷ_i
     +
     ∂ g(ŷ_i,ŷ_j,l_ij) /∂ŷ_j ∇_θŷ_j





























We can now express the gradient of the loss function of the mini-batch in matrix form as:

    ∇ _θ  L = [ [ ∇ _θŷ_1 ∇ _θŷ_2       … ∇ _θŷ_M;         ]]P1_M













where 1_M is the vector of all ones of length M. For a
standard single-branch network, we would average the gradients for all
batch samples to obtain the gradient of the mini-batch. This is
equivalent to setting P to the identity matrix
in Eq. <ref> above. For Siamese networks where we
consider all pairs in the mini-batch we obtain Eq. <ref> by
setting P to:


    =1.5pt
    P = [
        
          0    ∂ g(ŷ_1,ŷ_2,l_12) /∂ŷ_1    ⋯   ∂ g(ŷ_1,ŷ_M,l_1M)/∂ŷ_1 
    ∂ g(ŷ_1,ŷ_2,l_12) /∂ŷ_2     0    ⋯   ∂ g(ŷ_2,ŷ_M,l_2M) /∂ŷ_2 
    ⋮   ⋮   ⋱   ⋮
    ∂ g(ŷ_1,ŷ_M,l_1M) /∂ŷ_M    ⋯   ⋯    0 ]


The derivation until here is valid for any Siamese loss function of
the form Eq. <ref>. For the specific case of the ranking
hinge loss replace g in Eq. <ref> with
Eq. <ref> and obtain:

    P =  [  0     a _ 12    ⋯    a _ 1M 
     a _ 21     0    ⋯    a _ 2M 
    ⋮   ⋮   ⋱   ⋮
     a _ M1    ⋯    a _ M(M-1)     0 ],

where



















    a_ij =
    
    0    if  l_ij( ŷ_j -  ŷ_i ) + ε≤  0   
    
    l_ij   otherwise

and l_ij∈{ - 1,0,1} where 1 (-1) indicates
that y_i > y_j (y_i < y_j), and 0 that y_i = y_j
or that they cannot be compared as is the case with images
corrupted with different distortions.



 
 
 







The above case considered a single distortion. Suppose instead we have
D types of distortions in the training set. We can then compute the
gradient of the loss using a block diagonal matrix as:

    ∇ _θ  L = [ [ ∇ _θŶ^1 …∇ _θŶ^D;                  ]][ [ A^1   0   ⋯   0;   0 A^2   ⋯   0;   ⋮   ⋮   ⋱   ⋮;   0   0   ⋯ A^d;     ]]1_M




where

    A ^ d    =   [
    
    0     a _ 12 ^ d    ⋯    a _ 1M^d ^ d 
     a _ 21 ^ d     0    ⋯    a _ 2M^d ^ d 
    ⋮   ⋮   ⋱   ⋮
     a _ M^d1 ^ d    ⋯    a _ M^d(M^d-1) ^ d     0
    ], 
    ∇ _θŶ^d     =   [
    ∇ _θŷ_d1   ⋯   ∇ _θŷ_dM
    ].








where ŷ_ mn refers to the network output of the n^th image with the m^th
distortion, d ∈{1, 2, …, D}, and M = ∑M^d where
M^d is the number of images with distortion d in the mini-batch.
In the definition of A^d above, the a^d_ij are the gradient
coefficients as in
Eq. <ref>. 













 §.§ Fine-tuning for NR-IQA



After training a Siamese network to rank distorted images, we then
extract a single branch from the network for fine-tuning. Given M
images in mini-batch with human IQA measurements, we denote the
ground truth quality score of the i-th image as y_i, and the
predicted score from the network is ŷ_i, as above. We fine-tune
the network using squared Euclidean distance as the loss
function in place of the ranking loss used for the Siamese network:

    L(y_i, ŷ_i) =  1 / M ∑ _ i=1 ^ M  (y_i-ŷ_i)^2















§ EXPERIMENTAL RESULTS



In this section we report on a number of experiments designed to
evaluate the performance of our approach with respect to baselines and
the state-of-the-art in IQA. 



 §.§ Datasets


We use two types of datasets in our experiments: generic, non-IQA
datasets for generating ranked images to train the Siamese network,
and IQA datasets for fine-tuning and evaluation.

IQA datasets.  We perform experiments on two standard
IQA benchmark datasets.  The LIVE <cit.> consists of
808 images generated from 29 original images by distorting them with
five types of distortion: Gaussian blur (GB), Gaussian noise (GN),
JPEG compression (JPEG), JPEG2000 compression (JP2K) and fast fading
(FF). The ground-truth Mean Opinion Score for each image is in the
range [0, 100] and is estimated using annotations by 161 human
annotators.The TID2013 <cit.> dataset consists of
25 reference images with 3000 distorted images from 24 different
distortion types at 5 degradation levels. Mean Opinion Scores are in
the range [0, 9]. Distortion types include a range of noise,
compression, and transmission artifacts. See the original publication
for the list of specific distortion types.









Datasets for generating ranked pairs. To test on the
LIVE database, we generate four types of distortions which are widely
used and common : GB, GN, JPEG, and JP2K. To test on TID2013, we
generate 17 out of a total of 24 distortions (apart from #3,
#4,#12, #13, #20, #21, #24). For the distortions which we could
not generate, we apply fine-tuning from the network trained from the
other distortions. This was found to yield satisfactory results.
We use two datasets for generating ranked image pairs.  

The
Waterloo dataset consists of 4,744 high quality natural
images carefully chosen from the Internet. Additionally, we use the
validation set of the Places2 <cit.> dataset of 356
scene categories. There are 100 images per category in the validation
set, for a total 36500 images. After distortion, we have many more
distorted images for learning an image quality ranking embedding. The
aim of using this dataset is to demonstrate that high-quality ranking
embeddings can be learned using datasets not specifically designed for
the IQA problem.





















 §.§ Experimental protocols


We investigate a number of network architectures and use standard IQA
metrics to evaluate performance.






















Network architectures. We evaluate three typical network
architectures varying from shallow to deep. We refer to them as:
Shallow, AlexNet <cit.>, and
VGG-16 <cit.>. The shallow network has four convolutional layers and one fully connected layer.  For AlexNet
and VGG-16 we only change the number of
outputs since our objective is to assign one score for each
distorted image.
















Strategy for training and testing. We randomly sample
sub-images from the original high resolution
images. We do this instead of scaling to avoid introducing
distortions caused by interpolation or filtering. The size of sampled
images is determined by each network. However, the
large size of the input images is important since input sub-images
should be at least 1/3 of the original images in order to capture
context information. This is a serious limitation of the patch
sampling approach that samples very small 32 × 32 patches from
the original images. In our experiments, we sample 227 × 227
and 224 × 224 pixel images, depending on the network. We use the Caffe <cit.> framework and train using mini-batch Stochastic Gradient Descent (SGD) with an initial learning rate of 1e-4 for efficient Siamese network training and 1e-6 for fine-tuning. Training rates are decreased by a factor of 0.1 every 10K iterations for a total of 50K iterations. For both training phases we use ℓ_2 weight decay (weight 5e-4). During training we sample a single subimage from each training image per epoch. 

When testing, we randomly sample 30 sub-images from the original images as suggested in <cit.>. The average of the outputs of the
sub-regions is the final score for each image.

Evaluation protocols. Two evaluation metrics are
traditionally used to evaluate the performance of IQA algorithms: the
Linear Correlation Coefficient (LCC) and the Spearman Rank Order
Correlation Coefficient (SROCC). LCC is a measure of the linear
correlation between the ground truth and the predicted quality
scores. Given N distorted images, the ground truth of i-th image
is denoted by y_i, and the predicted score from the network is
ŷ_i. The LCC is computed as:

    LCC = ∑_ i=1 ^ N   (  y _ i - y  ) ( ŷ_ i -ŷ )/√(∑ _i^N (  y _ i - y  )^ 2 )√(∑ _ i ^ N  ( ŷ_ i -ŷ)^2  )

where  y and  ŷ are the means of the
ground truth and predicted quality scores, respectively.

Given N distorted images, the SROCC is computed as:

    SROCC = 1 -  6∑ _ i=1 ^ N (  v _ i - p _ i )  ^ 2 / N(  N ^ 2 -1 )  ,

where v_i is the rank of the ground-truth IQA score y_i in
the ground-truth scores, and p_i is the rank of ŷ_i
in the output scores for all N images. The SROCC measures the
monotonic relationship between ground-truth and estimated IQA.




















































 §.§ Learning NR-IQA from rankings

We performed a number of experiments to evaluate the ability of
Siamese networks to learn to capture image distortions from a large
dataset of image quality rankings.  In addition, we measure the
impact of the efficient Siamese backpropagation approach described in
section <ref>.












Siamese networks and IQA discrimination.  To
demonstrate the ability of our ranking networks to discriminate image
quality, we trained our Siamese network on the Places2 validation set
(without applying fine-tuning on IQA data) corrupted with five levels of a
single distortion. We then used that network to predict image quality
for synthetically-distorted images from the Waterloo dataset corrupted
using the same five levels of the same distortion. The network outputs
are plotted as histograms in Fig. <ref> for the JPEG distortion[Graphs for the other distortions are in the supplementary material.]. In the plot, we
divide the observations according to the true distortion level
(indicated by the color of the histogram). The model discriminates different levels of distortions on
Waterloo, even though the acquisition process and the scenes of the two
datasets are totally different.

Efficient Siamese backpropagation.  The objective of
this experiment is to evaluate the efficiency of our Siamese
backpropagation method.  We compare our method to both standard random
pair sampling, and a hard-negative mining method similar
to <cit.>.[We experimented with several
  hard-negative mining methods and found this to work best.]
   For standard random pair sampling we randomly choose 36
pairs for each mini-batch from the training sets. For the hard
negative mining strategy we start from 36 pairs in a mini-batch, and
gradually increase the number of hard pairs every 5000 iterations. For
our method we pass 72 images in each mini-batch. With these settings
the computational costs of all three methods is equal, since at each
iteration 72 images are passed through the network. We use AlexNet for
this experiment. The comparison of convergence rates on JPEG[Results for the other distortions are in the supplementary material.] is shown in Fig. <ref>. The efficient Siamese backpropagation not only converges much faster, but also converges to a considerably lower
loss.

Network performance analysis. Here we evaluate whether we can learn a useful image representation from large image ranking datasets. We randomly split on the original, high-quality images before distortion from the LIVE dataset
into 80% training and 20% testing samples and compute the average
LCC and SROCC scores on the testing set after training to
convergence. This process is repeated ten times and the results are
averaged. In Table <ref> we compare results for three different networks: Shallow, AlexNet and VGG-16. We obtain the best results with the VGG-16 network, which is also the deepest network. This indicates learning from ranking makes it possible to train very deep networks efficiently without overfitting. These results are obtained by training from scratch, however we found that initializing the weights with a network pre-trained on ImageNet further improved the results. In the remainder of the experiments we thus use the VGG-16 network initialized with a pre-trained weights to train the ranking network in the following experiments.








Baseline performance analysis. In this experiment, we evaluate the effectiveness of using rankings to estimate image quality. We compare tree methods: fine-tuning the VGG-16 network initialized from ImageNet to obtain the mapping from images to their predicted scores (called Baseline), our method to train VGG-16 (initialized from ImageNet) on ranking database using all ranking dataset we generate (called RankIQA), and finally our RankIQA approach fine-tuned on the TID2013 database after training using ranked pairs of images (called RankIQA+FT). 

We follow the experimental protocol used in HOSA <cit.>. The entire TID2013 database including all types of distortions is divided into 80% training images and 20% testing images according to the reference images. Thus, the same image can never appear in both training and test sets.  The results are shown in Table <ref>, where ALL means testing all distortions together. All the experiments are performed 10 times and the average SROCC is reported[LCC results are provided in supplementary material.].

From Table <ref>, we can draw several conclusions. First, it is hard to obtain good results by training a deep network directly on IQA data. This is seen in the Baseline results and is due to the scarcity of training data. Second, our RankIQA method achieves superior results on almost all individual distortions even without ever using the TID2013 dataset – which strongly demonstrates the effectiveness of training on ranking data. Slightly better results are obtained on ALL without comparing among different distortions during training the ranking network. The RankIQA-trained network alone does not provide accurate IQA scores (since it has never seen any) but does provide high correlation with the IQA scores as measured by SROCC.  After fine-tuning on the TID2013 database (RankIQA+FT), we considerably improve the ALL score, and improve the baseline by 16%. However, in the fine-tuning process to optimize the ALL score the network balances the various distortions, and results decrease for several distortions. 




 §.§ Comparison with the state-of-the-art


We compare the performance of our method using the VGG-16
network with state-of-the-art methods.  We perform experiments on the TID2013 and LIVE dataset.[Results on CSIQ <cit.> and MLIVE <cit.> are in supplementary material.]










Evaluation on TID2013. Table <ref> also includes results of state-of-the-art methods. We see that for several very challenging distortions (14 to 18), where all other methods fail, we obtain satisfactory results. For individual distortions, there is a huge gap between our RankIQA method and other IQA methods on most distortions. The state-of-the-art method HOSA performs slightly better than our methods on 6 out of 24 distortions. For all distortions, our method RankIQA+FT achieves about 5% higher than HOSA. Our methods perform well on distortions which are not included when training the ranking network, which indicates that different distortions share some common representation and training the network jointly on all distortions.

Evaluation on LIVE.  As done
in <cit.>, we randomly split the reference images on LIVE dataset
into 80% training samples and 20% testing, and compute the average
LCC and SROCC scores on the testing set after training to
convergence. This process is repeated ten times and the results are
averaged. These results are shown in Table <ref>. The best method
for each dataset is indicated in bold. 
The column indicated with ALL means we combine all
five distortions together on LIVE dataset to train and test the
model. For fair comparison with the state-of-the-art, we train our
ranking model on four distortions except FF, but we fine-tune our
model on all five distortions in the LIVE dataset to compute ALL. Our approach
achieves about 1% better than the best results reported on ALL
distortions for LCC. Similar conclusions are obtained for SROCC. This indicates that our method outperforms existing work
including the current state-of-the-art NR-IQA method SOM <cit.> and DNN <cit.>, and also state-of-the-art FR-IQA method DCNN <cit.>. To the best of our knowledge this is the
  first time that an NR-IQA method surpasses the performance of FR-IQA
  methods (which have access to the undistorted reference image) on all LIVE distortions using the LCC and SROCC evaluation method.



















































 §.§ Independence from IQA training data


This final experiment is to demonstrate that our framework can be also
trained on non-IQA datasets. In the previous experiment the network is
trained from high-quality images of the Waterloo dataset. Instead here
we use the validation set of the Places2 dataset to generate ranked
images in place of the Waterloo dataset. The Places2 dataset is of
lower quality than Waterloo and is not designed for IQA research.  As
in the previous experiment, the final image quality scores are
predicted by fine-tuning on the LIVE dataset.  The performance of this
model is compared with the results trained on Waterloo in
Table <ref>. The SROCC and LCC values are very similar,
demonstrating that our approach can be learnt from arbitrary, non-IQA
data.











§ CONCLUSIONS

To address the scarcity of IQA data we have proposed a 
method which learns from ranked image datasets. Since this data can be generated in abundance we can
train deeper and wider networks than previous work. In addition, we have proposed a method for
efficient backpropagation in Siamese networks which circumvents the
need for hard-negative mining. Comparison with standard
pair sampling and hard-negative sampling shows
that our method converges faster and to a lower loss.  Results on LIVE and TID2013 datasets
 show that our NR-IQA approach
obtains superior results compared to existing NR-IQA techniques and
even FR-IQA methods. 

Acknowledgements
We acknowledge  the  Spanish project TIN2016-79717-R, the CHISTERA project M2CR (PCIN-2015-251) and the CERCA Programme / Generalitat de Catalunya. Xialei Liu acknowledges the Chinese Scholarship Council (CSC) grant No.201506290018. We also acknowledge the generous GPU donation from NVIDIA.








ieee



