IEEE TRANSACTIONS ON MULTIMEDIA

11

Automatic Image Cropping for Visual Aesthetic
Enhancement Using Deep Neural Networks and
Cascaded Regression

arXiv:1712.09048v2 [cs.CV] 14 Jan 2018

Guanjun Guo, Hanzi Wang*, Senior Member, IEEE, Chunhua Shen, Yan Yan,
Yan, Member,
Member, IEEE,
IEEE,
Hong-Yuan Mark Liao, Fellow, IEEE

Abstract—Despite recent progress, computational visual aesthetic is still challenging. Image cropping, which refers to the
removal of unwanted scene areas, is an important step to improve
the aesthetic quality of an image. However, it is challenging to
evaluate whether cropping leads to aesthetically pleasing results
because the assessment is typically subjective. In this paper, we
propose a novel cascaded cropping regression (CCR) method
to perform image cropping by learning the knowledge from
professional photographers. The proposed CCR method improves
the convergence speed of the cascaded method, which directly
uses random-ferns regressors. In addition, a two-step learning
strategy is proposed and used in the CCR method to address the
problem of lacking labelled cropping data. Specifically, a deep
convolutional neural network (CNN) classifier is first trained
on large-scale visual aesthetic datasets. The deep CNN model
is then designed to extract features from several image cropping
datasets, upon which the cropping bounding boxes are predicted
by the proposed CCR method. Experimental results on public
image cropping datasets demonstrate that the proposed method
significantly outperforms several state-of-the-art image cropping
methods.
Index Terms—Image cropping, cascaded cropping regression,
convolutional neural network, random-ferns regressor.

I. I NTRODUCTION
OMPUTATIONAL image understanding consists of not
only image classification, object detection, object tracking and other popular computer vision tasks, but also semantic
inference of aesthetic from images. Computational inference
of aesthetic has a wide range of applications. Image aesthetics
can be used to recommend aesthetically pleasing images in an
image repository to users. General consumers or designers can
utilize the feedback from an automated aesthetic evaluation
system to improve decisions [1]. However, computational inference of image aesthetics remains a challenging task because
image aesthetics is highly subjective and difficult to represent
with precise mathematical explanations.
Although aesthetic measurement is subjective, some works
attempt to learn visual properties that make photographs

C

Corresponding author.
author. Tel./fax:
Tel./fax: +86
+86 5922580063.
5922580063.
** Corresponding
G. Guo,
Guo, H.
H. Wang
Wang and
and Y.
Y. Yan
Yan are
are with
with the
the Fujian
Fujian Key
G.
Key Laboratory
Laboratory of
of Sensing
Sensing
and Computing
Computing for
for Smart
Smart City,
City, and
and the
the School
School of
and
of Information
Information Science
Science and
and EnEngineering, Xiamen
Xiamen University,
University, Xiamen
Xiamen 361005,
361005, Fujian,
gineering,
Fujian, P.
P. R.
R. China.
China. (E-mail:
(E-mail:
gjguo@stu.xmu.edu.cn; hanzi.wang@xmu.edu.cn;
hanzi.wang@xmu.edu.cn; yanyan@xmu.edu.cn).
gjguo@stu.xmu.edu.cn;
yanyan@xmu.edu.cn).
C. Shen
Shen is
is with
with the
the Australian
Australian Center
Center for
for Visual
C.
Visual Technologies,
Technologies, and
and the
the
School of
of Computer
Computer Science
Science at
at The
The University
University of
School
of Adelaide,
Adelaide, SA
SA 5005,
5005,
Australia. (E-mail:
(E-mail: chunhua.shen@adelaide.edu.au).
chunhua.shen@adelaide.edu.au).
Australia.
H. Y.
Y. Mark
Mark Liao
Liao is
is with
with the
the Institute
Institute of
of Information
H.
Information Science,
Science, Academia
Academia
Sinica, Taipei
Taipei 115,
115, Taiwan.
Taiwan. (E-mail:
(E-mail: liao@iis.sinica.edu.tw).
liao@iis.sinica.edu.tw).
Sinica,

(a)

(b)

Fig. 1:
1: Illustration
Illustration of
of the
the proposed
proposed CCR
CCR method
method to
to crop
crop
Fig.
aesthetic
regions
in
two
images.
(a)
Input
images.
(b)
The
aesthetic regions in two images. (a) Input images. (b) The
cropped
images
obtained
by
the
proposed
method.
The
black
cropped images obtained by the proposed method. The black
regions are
are cut
cut off
off by
by the
the proposed
proposed method.
method.
regions

aesthetically beautiful
beautiful [2],
[2],[3],
[3],[4],
[4],[5],[5],
[6],[6].
[7].For
For example,
example,
in [2], color, texture and
and other
other low-level
low-level visual
visual features
features are
are
extracted for training aa two-class
two-class classification
classification tree
tree model
model
to predict the aesthetic
aesthetic quality
quality of
of an
an image.
image. Several
Several recent
recent
methods [3], [4], [7]
[8] have
have followed
followed this
this scheme.
scheme. Specifically,
Specifically,
different handcrafted features
features inspired
inspired by
by aesthetic
aesthetic experience
experience
are first extracted. Machine
Machine learning
learning models
models are
are then
then trained
trained to
to
identify the visual properties
properties of
of most
most relevance
relevance to
to the
the aesthetic
aesthetic
quality. As deep CNN [8]
[9] is
is successful
successful on
on many
many computer
computer
vision tasks, where features
features are
are automatically
automatically learned
learned from
from raw
raw
images, CNN has also been
been used
used on
on aesthetic
aesthetic categorization
categorization
for images [10],
[9], [10],
[11], [11].
[12]. Typically,
Typically, CNN
CNN requires
requires aa large
large
number of training samples
samples to
to alleviate
alleviate overfitting.
overfitting. A
A recently
recently
published large-scale dataset
dataset called
called the
the large-scale
large-scale aesthetic
aesthetic
visual analysis (AVA) dataset
dataset [12]
[13] makes
makes itit feasible
feasible to
to train
train aa
CNN based aesthetic classifier.
classifier. The
The task
task of
of blind
blind image
image quality
quality
assessment (BIQA) is closely
closely related
related to
to the
the task
task of
of aesthetic
aesthetic
categorization, and BIQA
BIQA methods
methods (such
(such as
as [13],
[14], [14])
[15]) do
do
not need many training samples,
samples, these
these BIQA
BIQA methods
methods usually
usually
focus high-quality image
image regions
regions rather
rather than
than object
object regions.
regions.
Thus, these BIQA methods
methods are
are usually
usually not
not appropriate
appropriate for
for the
the
task
of
image
cropping.
task of image cropping.

IEEE TRANSACTIONS ON MULTIMEDIA

Image cropping is a key step to generate aesthetically pleasing images. A distinction between a professional photographer
and an amateur heavily relies on whether one can remove
distracting contents and highlight desired subjects to enhance
the visual aesthetic of an image. Some composition rules for
obtaining aesthetically pleasing photographs, such as the rules
of thirds (i.e., an approximation of the Golden Ratio) and
the rules of odds have been developed. Inspired by these
composition rules, several image cropping methods, which are
usually based on handcrafted features, have been proposed.
Representative works are [16], [4]. However, the composition
rules may be different for various styles of images. For
example, portrait photographers often highlight a person while
landscape photographers focus more on the interaction among
the elements of an image. In Gestalt psychology [17], the
concept of goodness of configuration shows that the elements
in an image are not isolated. People prefer to choose patterns,
which have properties such as symmetry, simplicity, etc.
Therefore, image cropping for visual aesthetic enhancement
is a complicated and high-level cognition process.
In this paper, we develop a machine learning method to
automatically crop images to produce the cropped images
that are aesthetically pleasing. Specifically, a two-step learning
approach is proposed to solve the automatic cropping problem
with a small number of training samples that contain cropping
information. First, a CNN classifier is trained using a large
collection of visual aesthetic datasets. The CNN features
are then extracted based on the trained classifier. Using the
CNN features as the input, a cascaded cropping regression
(CCR) method, which combines a set of weak random-ferns
regressors [18] as a primitive regressor, is proposed to fit
the image cropping information annotated by professional
photographers. A key motivation of this two-step learning
approach is that the amount of image cropping data is limited
and directly fitting a cropping model using deep CNNs may
easily lead to overfitting. Bounding-box labels are very limited
and expensive, especially in the case where bounding-box
labels are annotated by professionals. However, a large number
of weakly labeled images, which have image-level aesthetic
labels, are available from the internet. Therefore, this work
presents a two-step learning approach to respectively learn a
CNN feature extractor and a set of regressors for the task
of image cropping. The advantages of the proposed two-step
learning approach are as follows. First, the proposed approach
leverages a large number of aesthetic image data, and thus can
effectively learn image features. Second, the proposed CCR
method is effective in selecting features and learning cropping
information in a cascaded manner. At each stage of regression,
CNN features are extracted from the cropping region obtained
at the previous stage. The final cropping result is obtained after
several stages of regression. The primitive regressor employed
in the CCR method is an ensemble of random-ferns regressors
selected by using the gradient boosting algorithm [19]. Each
primitive regressor indirectly selects features from the CNN
features by aggregating the features selected by the set of
random-ferns regressors. In contrast to conventional cascaded
methods, which directly use one random-ferns regressor as
the primitive regressor, the CCR method converges quickly

2

in a few stages, significantly reducing the computational
complexity.
To the best of our knowledge, this work is the first to
present a cascaded regression method with the CNN features
for the task of automatic image cropping. We improve the
convergence speed of the cascaded regression method based
on random-ferns regressors, which makes the proposed image
cropping method quite efficient. In addition, we propose a twostep learning strategy for limited labelled cropping data. The
proposed image cropping method is effective and significantly
outperforms several state-of-the-art image cropping methods.
Fig. 1 illustrates the proposed image cropping method for visual aesthetic enhancement. The proposed method can imitate
the manipulations of expert photographers to remove distracted
regions (such as watermarks, non-subject regions, etc.) and
highlight a subject in an image.
II. R ELATED W ORK
Existing image cropping methods can be roughly divided
into three categories. Methods of the first category are
attention-based, and output a cropping bounding box around
an informative object. The informative object can be a salient
object obtained by different saliency detection methods (such
as [20], [21]). For example, Marchesotti et al. [22] propose
a framework for visual saliency detection where one or
more thumbnails are extracted from the obtained saliency
maps. Thumbnails are usually salient foreground regions
while non-informative pixels become part of the background.
Fang et al. [23] also utilize a spatial pyramid of saliency maps
as a composition feature to force a cropped image to contain
a salient object. In addition, faces or other regions of interest
are often used as informative regions for image cropping [24],
[25], [26].
Methods of the second category rely on the aesthetic
evaluation of cropping results. Machine learning methods
are often used to identify the aesthetic of cropped images.
Moreover, these methods also consider the spatial distribution
of elements in an image to obtain an optimized arrangement of
the image elements. Representative works are [27], [16], [28].
However, aesthetic based methods are sensitive to evaluating
the attractiveness of a cropped image, and therefore these
methods focus on what remains in the cropped image. To
overcome the above problem, Yan et al. [4], [29] propose
an image cropping method belonging to the third category of
image cropping methods, namely an experience-based method.
In their method, they construct several cropping datasets,
which are annotated by three professional photographers,
for image cropping. Various handcrafted features are then
extracted for regressing the cropping values annotated by the
professional photographers. This method emphasizes professionals’ experience or the change caused by the manipulations
of image cropping. A drawback is that handcrafted features are
often limited and may lack useful information for high-level
computer vision tasks.
Inspired by the success of deep CNNs on various vision
tasks, in this paper we propose to learn the CNN features for
automatic image cropping. To alleviate overfitting, the CNN

IEEE TRANSACTIONS ON MULTIMEDIA

3

CCR

Input
Image

Update
Crop

Primitive
CPR
Regressor

R

Output
Crop

(a)

(b)

(c)

(d)

(e)

Fig. 2: The framework of the proposed CCR method. The dashed box denotes the procedure where T stages of regression are
performed for obtaining the final cropping result. (a) An input image (i.e., the initial cropping region). (b) The cropping-indexed
CNN features. (c) The proposed primitive regressor used to predict the values of the cropping region increment based on the
features obtained in (b). (d) Updating the cropping region. (e) The final cropping result.
model is trained on a combination of the large-scale AVA
dataset and the CHUKPQ dataset. The learned model is then
applied to several cropping datasets for extracting the CNN
features. With the extracted CNN features, we propose the
CCR (cascaded cropping regression) method to fit the cropping
information annotated by professional photographers.
III. T HE P ROPOSED CCR M ETHOD
This section provides the details of the proposed CCR
method for visual aesthetic enhancement. Fig. 2 shows the
framework of the proposed CCR method. The size of the
initial cropping region C 0 is set to be the same as that of
the original image I. The CNN features are extracted from
the cropping region C 0 by using a pre-trained CNN model
(the training details of the CNN model will be presented
in subsection III-A). Then a primitive regressor is used to
estimate an improved cropping region using the CNN features
at each stage. The final cropping result is obtained after T
stages of regression.
A cropping region C = [x̂1 , ŷ1 , x̂2 , ŷ2 ]T consists of the
coordinates of the top-left and bottom-right corners of a
rectangle region. Given an original image I, the goal of image
cropping is to estimate a cropping region C that is as close as
possible to the ground-truth cropping region Ĉ provided by a
professional photographer. The above problem can be modeled
as the least-squares regression problem as follows:
C̄ = argmin kC − Ĉk2 .

(1)

C

This least-squares regression problem1 is solved by the proposed CCR method. Before describing the proposed CCR
method in detail, we propose a CNN feature extraction strategy
in the next subsection.
A. Training an Aesthetic Classifier with CNN for Feature
Extraction
Although some handcrafted features (such as exclusion
and compositional features in [4], [29]) for image cropping
have been proposed, these features fail to cover all possible
situations for image cropping. Moreover, visual aesthetic assessment is subjective. Therefore, instead of designing some
1 The least-squares loss function shown in Eq. (1) can be improved by using
the other loss functions (such as [30] and [31]).

handcrafted features, the proposed method directly learns
features from large-scale visual aesthetic datasets based on
the deep convolutional neural network (CNN). The AVA
dataset [13] and the CUHKPQ dataset [32] are two annotated
large-scale visual aesthetic datasets, where most images are
from the photography website of Dpchallenge2 . The images
of the two datasets received a number of votes or aesthetic
judgements from the members of the photography community.
As high-quality visual pleasing images usually have a clear
topic and an impressive composition, the features learned
from these images are relatively more useful for enhancing
visual aesthetic during the image cropping process. Note that
a cropping regressor cannot be directly trained using CNN on
these two datasets due to the lack of cropping annotation.
We design a deep CNN structure (as shown in Fig. 3) to
extract valid features for image cropping. The first convolutional layer is used to filter input images with 32 kernels of size
5 × 5 × 3 and it outputs 32 feature maps, which have the same
size as each of the input images. Each convolutional layer
also outputs 32 feature maps. Each convolutional layer is then
activated by a rectified linear unit (ReLU) [33] and followed by
a max-pooling layer in the first four layers. The max-pooling
layer partitions the feature maps from the previous layer into
a set of 2 × 2 non-overlapping neighborhoods, and outputs the
maximum value for each neighborhood. The last layer is the
spatial pyramid pooling layer (SPP) [34], which partitions each
input feature map into divisions from a fine level to a coarse
level and aggregates the local features in each division. Using
the SPP layer can alleviate the problem that resizing a visually
pleasing image may potentially damage its aesthetic. In total,
five convolutional layers, four max-pooling layers and one SPP
layer are used in the designed network structure (as shown in
Fig. 3). The CNN model corresponding to this structure is
trained on the combined AVA and CUHKPQ datasets.
Training datasets. The aesthetic classifier is trained using
the combined AVA and CUHKPQ dataset. The AVA dataset
contains more than 250,000 images collected from the Dpchallenge website. Each image has about 210 aesthetic ratings
ranging from 1 to 10. We divide these images into two
categories (i.e., low-quality images and high-quality images)
for training a two-class CNN model. Following the same
strategy used in [13], [10], a parameter δ is used to discard
2 http://www.dpchallenge.com

IEEE TRANSACTIONS ON MULTIMEDIA

4

Convolutional layer 1

5

2
2

5

32
Input layer

Convolutional
layer 2

5

5

2

5

32
Max-pooling
layer 1

Convolutional
layer 3

2

32

2
2

5
32
Max-pooling
layer 2

32

Convolutional
layer 4
5

2
2

5
32
32
Max-pooling
layer 3

Convolutional
layer 5
5

Soft-max
layer

2
2

5
32
32
Max-pooling
layer 4

32
SPP layer

Fig. 3: The proposed deep CNN structure, which is trained using the combined AVA and CHUKPQ datasets, is used to extract
the CNN features.
ambiguous images from the training set. The images with
average scores smaller than 5−δ are referred to as low-quality
images. The images with average scores larger than or equal
to 5 + δ are considered as high-quality images. The images
with average scores between 5 − δ and 5 + δ are considered
as ambiguous and are thus discarded. In the implementation,
we set the value of δ to be 1. Thus, 49,682 high-quality
images and 7,983 low-quality images are selected from the
AVA dataset. The CHUKPQ dataset contains about 30,000
images, which are collected from a variety of photography
websites. Each image in this dataset has been labelled as
either low or high quality. There are totally 10,525 highquality images and 19,167 low-quality images labelled in
the CHUKPQ dataset. The images from the two datasets are
combined according to their categories. In total, 60,207 highquality images and 27,150 low-quality images are collected
from these two datasets. To alleviate the class imbalance
problem, low-quality images are augmented by flipping each
low-quality image horizontally to obtain 54,300 low-quality
images in total. All the obtained images are split into 894
batches, and each batch consists of 128 images.
Training the aesthetic classifier. In this paper, we use the
GPU implementation code of T orch7 [35] to train the CNN
model. 880 batches are used for training and 14 batches are
used for testing. The whole training process takes around one
day on a computer equipped with two GTX-1080 GPUs, and
the trained CNN achieves 75.1% classification accuracy in test.
This result is competitive even compared with that obtained
by the state-of-the-art aesthetic classification method [10]. A
part of the filters in the trained model are shown in Fig. 4.
Most filters in the first convolutional layer are related to color.
Fig. 5 shows the feature maps obtained by the CNN model on
the image shown in the first row of Fig. 1.
The trained CNN model is then applied on each image in the
cropping datasets. The feature maps before the classifier layer
(i.e., the feature maps in the SPP layer in Fig. 3) are obtained
and used for cropping regression. In our implementation, the
SPP layer has three levels, where the feature maps of these
three levels are respectively partitioned into 2 × 2, 3 × 3 and
4×4 divisions. An input image becomes feature maps with the
size of (2×2+3×3+4×4)×32 after 5 layers of pooling. Thus,
the dimension of the obtained features is 928. Let t denote

the index of the stage in the proposed CCR method. In the tth
stage of regression, the CNN features xti are extracted from the
cropped image obtained from the (t − 1)th stage of regression.
We term these CNN features as the cropping-indexed CNN
features. Such features correlate with spatial positions and
also significantly improve the cascaded regression. However,
because the number of training samples for anesthetic image
cropping is often limited, existing regressors (such as CNN and
random-fern regressors) cannot be used directly in this task.
In the following subsection, we propose a novel and effective
regressor to address the problem caused by a limited number
of training data for the task of image cropping.

B. Learning a Primitive Regressor using Gradient Boosting
Algorithm
Inspired by the cascaded regression methods used in the
tasks of pose regression [18] and face alignment [36], a
cascaded regressor Rj = (rj1 , · · · , rjT ) is trained to output
Cj , where j (1 ≤ j ≤ 4) denotes the index of a 4-tuple
coordinate vector for a cropping region. For convenience, each
regressor rjt is called as a primitive regressor. T is the number
of primitive regressors used for training. Note that the 4-tuple
coordinate vector consists of the coordinates of the top-left
and bottom-right corners (i.e., [x̂1 , ŷ1 , x̂2 , ŷ2 ]) of a rectangle
region. Based on an initial cropping region C 0 and the
obtained cropping-indexed CNN features xt , each primitive
regressor rjt outputs a cropping region increment δCjt (defined
as Yj − Cjt−1 ), where Y is the 4-tuple coordinate vector for
the cropping region annotated by a professional photographer.
The estimates of the cascaded primitive regressors for image
cropping are computed by accumulating the predicted values
obtained by all the primitive regressors and the coordinates of
an initial crop. The above process can be written as:
Cj = Cj0 +

T
X

rjt (xt ).

(2)

t=1

Given N training samples (Ii , Yi )N
i=1 , the primitive regressors
(rjt (xt ))Tt=1 are sequentially learned until the test error stops
decreasing. Each primitive regressor rjt (an abbreviation of

IEEE TRANSACTIONS ON MULTIMEDIA
IEEE TRANSACTIONS ON MULTIMEDIA

5
5

Fig.
the proposed
proposed CNN
CNN model
model for
for image
image aesthetic
aesthetic classification.
classification.
Fig. 4:
4: The
The trained
trained filters
filters of
of the
the first
first convolutional
convolutional layer
layer of
of the

(a)

(b)

Fig. 5:
5: The
The feature
feature maps
maps obtained
obtained by the proposed CNN model on the image shown in the first
Fig.
first row
row in
in Fig.
Fig. 11 (a).
(a). (a)-(b)
(a)-(b) The
The
feature maps
maps obtained
obtained in
in the
the first and second convolutional layers of the CNN model, respectively.
feature
respectively.
t tt
(x ))
)) is
is learned
learned by
by minimizing
minimizing the
the following
following objective
objective
rrjtj(x
function:
function:
N
X
N
X
t−1
t
tj = argmin
r
kYij −
− (C
(Ct−1
+ rrj (x
(xtti))k
))k22,,
(3)
ij +
rj = argmin
kY
(3)
ij
j
i
ij
rj

rj

i=1
i=1

where Y denotes the jth value in the 4-tuple coordinate
where Yijij denotes the jth value in the 4-tuple coordinate
vector of the cropping region annotated by a professional
vector of the cropping region annotated by a professional
photographer for the ith image.
photographer for the ith image.
The choice of primitive regressors is diverse and it mainly
The choice of primitive regressors is diverse and it mainly
depends on what the application domain is. For example,
depends on what the application domain is. For example,
in [36], CNN is chosen as a primitive regressor to regress
in [37], CNN is chosen as a primitive regressor to regress
the coordinates of facial points. However, CNN is hard to

be
for theoftask
of image
of hard
limited
the applied
coordinates
facial
points. cropping
However,because
CNN is
to
labelled
cropping
data.
In
the
task
of
pose
regression
[17],
be applied for the task of image cropping because of limited
the
random-ferns
shows
betterregression
generalization
labelled
croppingregressor,
data. In which
the task
of pose
[18],
ability
than
the
tree
regressor
[37],
is
proposed
as
the primitive
the random-ferns regressor, which shows better generalization
regressor.
the experimental
results as
in the
Section
IV
ability thanHowever,
the tree regressor
[38], is proposed
primitive
show
that
the
random-ferns
regressor
is
too
weak
for
the
task
regressor. However, the experimental results in Section IV
of
cropping
Therefore,
in is
this
show
that theregression.
random-ferns
regressor
toopaper,
weak we
for propose
the task
to
learn
the
primitive
regressor
by
using
a
group
of
of cropping regression. Therefore, in this paper, werandompropose
ferns
regressors
to strengthen
its by
regression
The
gradito learn
the primitive
regressor
using a ability.
group of
randoment
boosting
algorithm
[18]
is
employed
to
implement
this.
ferns regressors to strengthen its regression ability. The gradiAccording
to
the
gradient
boosting
algorithm
[18],
a
nonent boosting algorithm [19] is employed to implement this.
parametric primitive regressor P can be obtained by using

IEEE TRANSACTIONS ON MULTIMEDIA

6

Algorithm 1 The Least-Squares Gradient Boosting Algorithm
0
1: Input: Data(xi , yi )N
i=1 ; The number of iterations M ; Initial value g = 0

2: for m = 1, 2, . . . , M do
m−1 (x )
3:
em
// compute the residual error
i
i = yi − g
P
m
4:
(ρm , αm ) = argminα,ρ N
i=1 [ei − ρh(xi ; α)] //select a randomferns regressor hm whose parameter is αm
5:
g m (xi ) = g m−1 (xi )+hm (xi ; αm )
//update the predicted value
6: end for
1
M
7: Output: P = (h , · · · , h )

Algorithm 2 The Cascaded Cropping Regression Algorithm
1: Input: Data(Ii , Yi )N
i=1 ; The number of stages T ; Initial cropping region
(Ci0 )N
i=1
2: for t = 1, 2, . . . , T do
3:
xti = f (Cit−1 , Ii )
// compute the cropping-indexed CNN features
4:
5:
6:
7:
8:
9:
10:

for j = 1, 2, . . . , 4 do
t = Y − C t−1
ỹij
// compute the residual error
ij
ij
t ) as the input by
Learning the primitive regressor rjt with (xti ,ỹij
using Algorithm 1
t = C t−1 + λ r t (xt )
Cij
//update the cropping region
j i
ij
end for
end for
Output: Rj = (rj1 , · · · , rjT )4j=1

According to the gradient boosting algorithm [19], a nonparametric primitive regressor P can be obtained by using
the least-squares gradient boosting algorithm, which is given
in Algorithm 1.
In Algorithm 1, g m (xi ) denotes the predicted value for the
ith sample in the mth iteration. h(xi ; α) denotes the randomferns regressor and α denotes its parameter. xi denotes the
features for the ith image. P = (h1 , · · · , hM ) is the obtained
primitive regressor, where M is the number of the randomferns regressors and hm is the random-ferns regressors with
αm as its parameter (i.e., hm (xi ; αm )). yi is the given label
value for the ith image and ρm denotes the best gradient
descent step-size. In the implementation of the random-ferns
regressor, dozens of random-ferns regressors h(xi ; α) are
generated uniformly in the mth iteration and the random-ferns
regressor hm (xi ; αm ) with the best performance is chosen.
Thus, the gradient descent step-size ρm is an indicator vector
with each element corresponding to one random-ferns regressor, where the value of an element is one if the corresponding
random-ferns regressor is chosen, and zero otherwise. The
gradient boosting algorithm uses a weak regressor h(xi ; α)
to predict the residual error value em
i between the predicted
value g m−1 (xi ) obtained from the (m − 1)th iteration and the
given label yi . The obtained regressor P is also called as the
master regressor consisting of M random-ferns regressors. For
more details, please refer to [19].
C. Cascaded Cropping Regression
This subsection describes the proposed CCR method in
detail based on a set of random-ferns regressors. A randomferns regressor is derived from a random-ferns classifier,
which is originally proposed for classifying keypoints in [39].
Assume that the number of weak classifiers in a random-ferns
classifier is M , the random-ferns classifier randomly partitions

Fig. 6: An example of the image cropping dataset provided
by [4], [29]. Three cropping regions are respectively annotated
by three professional photographers for each image in the
dataset. The three cropping regions are respectively shown in
three different colors.

Fig. 7: An example of the image cropping dataset provided
by [23]. Ten cropping regions are respectively annotated by ten
professional photographers for each image in the dataset. The
ten cropping regions are respectively shown in ten different
colors.
F -dimensional features into M groups and the size of each
group is S, where S is a small integer. Each weak classifier
takes an S-dimensional feature vector in the S-dimensional
feature space (instead of the original F -dimensional space)
as the input. Thus, the total number of the parameters of
each weak classifier is significantly reduced. The randomferns regressor is proposed in [18] for pose regression. Each
random-ferns regressor also takes an S-dimensional feature
vector as an input and yields a real output value. However, as
mentioned in Section III-B, S-dimensional features are weak
for the task of cropping regression. Thus, we use a group
of random-ferns regressors to learn the primitive regressor
by using the gradient boosting algorithm. Each random-ferns
regressor still randomly takes an S-dimensional feature vector
as an input, but the dimension of the features taken by the
obtained primitive regressor is more than S. Each primitive
regressor indirectly selects features from the F -dimensional
cropping-indexed CNN features by aggregating the features,

IEEE TRANSACTIONS ON MULTIMEDIA

which are selected by the group of random-ferns regressors.
Since each random-ferns regressor randomly selects an Sdimensional feature vector, some features may be selected
repeatedly. As a result, the total number of features selected
by the primitive regressor may vary. Based on the obtained
primitive regressor, four CCR models are trained to fit the
values of the 4-tuple coordinate vectors corresponding to the
cropping regions annotated by professional photographers,
respectively. The whole procedure of the proposed CCR
method is summarized in Algorithm 2. In order to reduce the
computational complexity, the cropping-indexed CNN features
are extracted in the outside loop and they are kept unchanged
in the inner loop.
In Algorithm 2, in order to alleviate the problem of scale
variations for different images, the coordinate values of the
initial cropping region Ci0 for the ith image are normalized
with the maximum size of images. f (Cit−1 , Ii ) denotes the
procedure of extracting the cropping-indexed CNN features
xti from the cropping region Cit−1 for the ith image Ii at the
t
tth stage. Then the residual error ỹij
, which is the difference
between the jth cropping value of the cropping region obtained
at the (t − 1)th stage and that of the cropping region annotated
t
by a professional photographer, is computed. With (xti ,ỹij
)
as the input, the primitive regressor rjt is learned by using
t
Algorithm 1. Then, the jth value of the cropping region Cij
at the tth stage is updated with the predicted value obtained
from rjt . The whole training process of CCR stops at the T th
stage. After the training process, the obtained CCR model is
used to predict a cropping region in an additive manner (see
Eqn. (2)).
The proof for the convergence of the proposed method
is readily derived from [19], [40], [18]. Let the relative
P error obtained
P by the primitive regressor P be γ =
( i d(r(xi ), yi )/ i d(y, yi )), where y denotes the single
uniform prediction and d(·,·) denotes a distance function. The
convergence of the proposed CCR method requires that γ < 1,
which means that the primitive regressor P predicts a better
result than the single uniform prediction.
Regularization and Shrinkage. To prevent the proposed
CCR method from overfitting, three regularization parameters
are introduced in our case. As shown in [19], a regularization
parameter for the gradient boosting models is the number of
weak learners or regressors (i.e., the number of the randomferns regressors M in CCR). The best value of M can be
estimated by evaluating the performance of the CCR method
on an independent validation set. In addition, controlling
the value of T is equivalent to control the training process
to stop at the T th stage. Thus, T is also a regularization
parameter. The third regularization parameter is the shrinkage
parameter, which often yields superior results to those obtained
by restricting the number of regressors. In Algorithm 2, the
introduction of the parameter λ in line 7 is a straightforward
shrinkage strategy for scaling the update at different stages.
IV. E XPERIMENTS
In this section, we evaluate the performance of the proposed
image cropping method for visual aesthetic enhancement. The

7

experiments include two parts. The first part evaluates the
performance of the proposed method with different parameter
settings. The second part evaluates the proposed method for
image cropping and compares it with several state-of-theart methods on cropping datasets. We use an image dataset,
which contains 950 images collected by Yan et al. [4], [29]
from the CHUKPQ dataset, for evaluation. The image dataset
contains seven classes of images, i.e., animal, architecture,
human, landscape, night, plant and static. A cropped region
is respectively annotated for each image by three professional
photographers, from which three cropping datasets are formed.
Fig. 6 shows an example of cropping regions annotated by the
three professional photographers, where the regions outside of
the bounding boxes are removed. Note that the images in the
CHUKPQ dataset, which are the same as those in the cropping
dataset, are removed in the process of training the aesthetic
classifier.
We note that [23] contributes a new cropping dataset
where each image has ten bounding boxes annotated by
ten professional photographers. However, the ten annotated
bounding boxes only have small overlaps (see Fig. 7 for
an example). It shows that there is little correlation among
the ten professional photographers’ knowledge or preference.
Therefore, the proposed CCR method is not evaluated on that
dataset.
Following the evaluation criteria used in [4], [29], the
Intersection over Union (IoU) metric and the Boundary Displacement Error (BDE) metric are
T adopted. The IoU metric
W
Wm
where Wp and Wm
is formulated as IoU = Wpp S Wm
respectively denote a cropping window annotated by a professional photographer and a cropping window generated by an
evaluated method on a test image. The BDE metric measures
the Euclidean distance of the generated cropping boundaries
Bm from those annotated by the photographer Bp , which is
kB −B k2
written as BDE = p 4 m .
A. Influence of the Parameters
After training CNN, the obtained CNN model is applied to
extract cropping-indexed CNN features for training a CCR
model. Several parameters are used for training the CCR
model. In this subsection, we examine how these parameters
affect the performance of the CCR method for image cropping.
Influence of the number of random-ferns regressors.
Fig. 8 (a) shows the IoU curves obtained by the proposed CCR
method with different numbers of random-ferns regressors.
The convergence speed of the proposed CCR method is closely
related to the number of random-ferns regressors. Usually,
using more random-ferns regressors to learn a primitive regressor leads to faster convergence speed in the proposed CCR
method. However, using too many random-ferns regressors to
learn a primitive regressor may lead a CCR model to overfit
a training dataset.
Using different primitive regressors. To compare the
convergence speed between the proposed CCR method and
the cascaded method (i.e., the CPR method in [18]), which
uses a single random-ferns regressor as the primitive regressor,
we apply the CPR method to the task of image cropping and

IEEE TRANSACTIONS ON MULTIMEDIA
IEEE TRANSACTIONS ON MULTIMEDIA
IEEE TRANSACTIONS ON MULTIMEDIA

0.8
0.8 0
0

20
20
#Stages
#Stages

40
40

(a)

1
1

CCR−−
CCR
CCR
CCR

C0
C10 0
12 C0
C
21 C 0
14 0
C
4

0.5
0.5

10
20
10
20
#Stages
#Stages

30
30

0
00
0

10
20
10
20
#Stages
#Stages

(b)

0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.2 0
0

IoU
IoU

0.9
0.9
0.88
0.88
0.86
0.86
0.84
0.84
0.82
0.82 0
0

IoU
IoU

0.85
0.85

1
15
510
10
20
20
40
40
80
80

IoU
IoU

IoU
IoU

0.9
0.9

8
8
8

30
30

(c)

AlexNet
AlexNet
Ours
Ours

10
20
10
20
#Stages
#Stages

30
30

(d)

(a) curves obtained by the proposed
(b)
(c)
Fig. 8: The IoU
method with different hyper-parameter
settings. (a) The(d)IoU curves with
Fig. 8:
8: The
The IoU
IoU curves
curves obtained
obtained by
by the
the proposed
proposed method
method with
with different
different hyper-parameter
hyper-parameter settings.
settings. (a)
(a) The
The IoU
IoU curves
with
Fig.
curves
with
different numbers of random-ferns regressors. (b) The comparison of the convergence speed between the proposed
CCR
method
different
numbers
of
random-ferns
regressors.
(b)
The
comparison
of
the
convergence
speed
between
the
proposed
CCR
method
− of random-ferns regressors. (b) The comparison of the convergence speed between the proposed CCR method
different
numbers
and the CCR
method. (c) The IoU curves obtained by the proposed method with different initial cropping parameters. (d)
and the
the CCR
CCR−− method.
method. (c)
(c) The
The IoU curves
curves obtained
obtained by
by the
the proposed
proposed method
method with
with different
different initial
initial cropping
cropping parameters.
and
parameters. (d)
(d)
The IoU
curves obtained
by theIoU
proposed method
with different
pre-trained CNN
models.
The
IoU
curves
obtained
by
the
proposed
method
with
different
pre-trained
CNN
models.
The IoU curves obtained by the proposed method with different pre-trained CNN models.

(a)

(b)

(c)

(d)

(e)

(f)

(a)
(b) the proposed CCR
(c) method can alleviate
(d) the problem of boundary
(e)
(f) show the
Fig. 9: Examples
showing that
cutting. (a)-(f)
Fig.
9: Examples
showing
that theCCR
proposed
CCR
method
can atalleviate
the
problem
of boundary
cutting.
(a)-(f)
show the
results
obtained
by
the
proposed
method
on
three
images
the
1st,
5th,
10th,
15th,
20th
and
30th
stage,
respectively.
Fig. 9: Examples showing that the proposed CCR method can alleviate the problem of boundary cutting. (a)-(f) show the
results
obtained
by
the
proposed
CCR
method
on
three
images
at
the
1st,
5th,
10th,
15th,
20th
and
30th
stage,
respectively.
results obtained by the proposed CCR method on three images at the 1st, 5th, 10th, 15th, 20th and 30th stage, respectively.

CCR− method is in that they use different primitive regressors.
−
CCR
method
is in8that
usethat
different
primitive
regressors.
As shown
Fig.
(b),they
the
convergence
speed
the
proposed
compare
itsinperformance
with
obtained
byofthe
proposed
As
shown
in
Fig.
8
(b),
the
convergence
speed
of
the
proposed
CCR
method
is
significantly
faster
than
that
of
the
CCR−
CCR method. In the implementation, the pose-indexed features
−
CCR
method
is
significantly
faster
than
that
of
the
CCR
method.
The
proposed
method
converges
in
only
about
30
in the CPR method are replaced by the cropping-indexed CNN
method.
The
proposed
method
in only
about
stages, indicating
that
the
proposed
CCR
method
extracts
− 30
features.
We call
this
variant
of theconverges
CPR
method
as only
CCR
. The
stages,
indicating
that
the
proposed
CCR
method
only
extracts
the
cropping-indexed
CNN
features
around
30
times
to
obtain
only difference between the proposed CCR method
and
the
−
thecropping-indexed
final
cropping
Infeatures
contrast,
the primitive
CCR
method
only
the
CNN
around
30 times
to obtain
−
CCR
method
is in result.
that
they
use
different
regressors.
−
obtains
a insmall
in average
IoU
inonly
the
the
final cropping
result.
Inconvergence
contrast,
thespeed
CCR
method
As
shown
Fig. 8improvement
(b), the
of values
the proposed
first
30
stages.
Thus,
the
proposed
CCR
method
significantly
obtains
a
small
improvement
in
average
IoU
values
in
the
CCR method is significantly faster than that of the CCR−
improves
the convergence
speed when
it uses
randomfirst
30 stages.
Thus, themethod
proposed
CCR
method
significantly
method.
The
proposed
converges
inmultiple
only
about
30
ferns regressors
as a primitive
regressor.
improves
the
convergence
speed
when
it
uses
multiple
randomstages, indicating that the proposed CCR method only extracts
Using
different
cropping
values. The effect of the
ferns
regressors
as a initial
primitive
regressor.
the cropping-indexed
CNN features
around 30 times to obtain
coordinates
of an initial
initial cropping
region
onThe
the
performance
Using
different
cropping
values.
the
− effect
the final cropping result. In contrast, the CCR
methodofonly
of the proposed
CCR cropping
method for
image
cropping
is also
coordinates
of
an
initial
region
on
the
performance
obtains a small improvement in average IoU values in the
investigated.
Each
initial
cropping
region cropping
is simplyisscaled
of
CCR
forCCR
image
also
firstthe
30 proposed
stages. Thus,
themethod
proposed
method significantly
down to 50%Each
and 25%
of cropping
its originalregion
size without
changing
its
investigated.
initial
is
simply
scaled
improves the convergence
when it uses multiple random1speed
1 0
0
C
and
C
in
Fig.
8
(c)),
respectively.
Then
centroid
(i.e.,and
down
to
50%
25%
of
its
original
size
without
changing
its
2 as a primitive
4
ferns regressors
regressor.
0 repeated
the experiment
other parameters
and 14 C 0 again
in Fig.while
8 (c)),therespectively.
Then
centroid
(i.e., 12 Cis
are
fixed.different
The isexperimental
results
are the
shown
Fig. of8 the
(c).
Using
initial cropping
values.
The in
effect
the
experiment
repeated
again
while
other
parameters
Asfixed.
we can
seeexperimental
although
the region
initial
cropping
parameters
coordinates
of
anthat
initial
cropping
on theinperformance
are
The
results
are shown
Fig.
8 (c).
different,
final
cropping
results
are cropping
almost
the
ofarewe
the
proposed
CCR
method
image
is same
also
As
can
see the
that
although
thefor
initial
cropping
parameters
after
several Each
stages
of cropping
regression.
The
proposed
CCR
investigated.
initial
cropping
region
simply
scaled
are
different,
the
final
results
are is
almost
themethod
same
obtains
the stages
highest
values
when
the
size CCR
of
the
initial
down
to 50%
and 25%
of its
original
without
changing
its
after
several
ofIoU
regression.
Thesize
proposed
method
1 0is set to
1 be
0 the same as that of the original
cropping
region
centroid
(i.e.,
C
and
C
in
Fig.
8
(c)),
respectively.
Then
obtains
the
highest
IoU
values
when
the
size
of
the
initial
2
4
cropping region is set to be the same as that of the original

image. The reason is that the proposed CCR method mainly
image.
The
is
proposed
method
mainly
focuses
on reason
cutting
outthat
thethe
regions
fromCCR
the other
current
window,
the
experiment
is repeated
again
while
the
parameters
focuses
on
cutting
out
the
regions
from
the
current
window,
and
more
effective
features
can
be
selected
by
the
proposed
are fixed. The experimental results are shown in Fig. 8 (c).
and
more
effective
selected
by
theparameters
proposed
primitive
when
thecan
initial
cropping
region
is larger
As
we
canregressor
see thatfeatures
although
thebeinitial
cropping
primitive
regressor
when
the
initial
cropping
region
is
than
the
cropping
region
to
be
estimated.
In
contrast,
the
are different, the final cropping results are almost the larger
same
than
the
cropping
region
to
be
estimated.
In
contrast,
the
proposed
CCR
method
may
infer
image
content
outside
of
after several stages of regression. The proposed CCR method
the
current
cropping
box
when
the
initial
cropping
region
proposed
CCR
method
may
infer
image
content
outside
of
obtains the highest IoU values when the size of the initial
is smaller
than
thesetcropping
region
estimated,
which
the
current
cropping
box
the to
initial
cropping
region
cropping
region
is
to bewhen
the
same
asbethat
of the original
makes
the
proposed
CCR
method
less
effective.
In
addition,
is
smaller
than
the
cropping
region
to
be
estimated,
which
image. The reason is that the proposed CCR method mainly
the coordinates
an the
initial
cropping
region
significantly
makes
the
CCR
method
less
In addition,
focuses
on proposed
cuttingofout
regions
fromeffective.
the
current
window,
affect
the convergence
speed
of
the proposed
CCR
method.
the
coordinates
of
an
initial
cropping
region
significantly
and more effective features can be selected by the proposed
If the the
average
IoU value
at of
the the
initial
stage is
small,
then
affect
convergence
speed
proposed
CCR
method.
primitive regressor when the initial cropping region is larger
thetheproposed
CCR
method
has
ainitial
fast convergence
speed
to
If
average
IoU
value
at
the
stage
is
small,
then
than the cropping region to be estimated. In contrast, the
increase
the average
IoU value.
The
differences
in convergence
the
proposed
CCR
method
has
a
fast
convergence
speed
to
proposed CCR method may infer image content outside of
speeds of
proposed method
withdifferences
different initial
cropping
increase
thethe
average
value.
The
in convergence
the current
croppingIoU
box
when
the initial cropping
region
parameters
therefore
become
smaller
in the latter
stages.
speeds
of
the
proposed
method
with
different
initial
cropping
is smaller than the cropping region to be estimated, which
Boundary
cuttingbecome
problem.
If the in
original
initial
cropping
parameters
therefore
smaller
the latter
stages.
makes the proposed
CCR method
less effective.
In
addition,
region
is scaled
downproblem.
to 50% orIf25%
of its original
size,
then
Boundary
cutting
the
original
initial
cropping
the coordinates of an initial cropping region significantly
the cropping
boundaries
mayorpass
through
an object,
which
region
is
scaled
down
to
50%
25%
of
its
original
size,
then
affect the convergence speed of the proposed CCR method.
causes
an unpleasant visual
effect.
However,
the proposed
the
cropping
passinitial
through
an isobject,
If the
averageboundaries
IoU valuemay
at the
stage
small,which
then
methodancan
alleviate visual
this problem
after several
stages of
causes
unpleasant
effect.
However,
the
the proposed CCR method has a fast convergence proposed
speed to
regression.
As
shown this
in Fig.
9, the after
boundaries
ofstages
the tree
method
can average
alleviate
problem
several
of
increase the
IoU value.
The differences
in convergence
are cut in As
the shown
beginning
stage9,due
toboundaries
the small of
sizetheof tree
the
regression.
in
Fig.
the
speeds of the proposed method with different initial cropping
original
cropping region.
However,
proposed
are
cut ininitial
the beginning
stage due
to the the
small
size ofCCR
the
original initial cropping region. However, the proposed CCR

0.45

0.3

0.4

0.2
0.1
0
0

Train
Test
50

100
#Epochs

150

Train
Test

0.86

0.35
0.3
0.25
0

(a)

0.88

IoU

IoU

IoU

0.4

P1
P2
P3

0.84
0.82

50

100
150
#Epochs

200

0.8
0

(b)

Fig. 10: The IoU curves obtained by a CNN regressor with Fig. 11: Th
Fig.
10: The IoU curves obtained by a CNN regressor with
different parameter initialization strategies on the training the three im
different
strategies
on bythethetraining
and test parameter
datasets. (a)initialization
The IoU curves
obtained
CNN extraction
and
test
datasets.
(a)
The
IoU
curves
obtained
by the
regressor whose parameters are randomly initialized.
(b) CNN
The proposed m
regressor
whose
parameters
are randomly
(b) the
The (b) The Io
IoU curves
obtained
by the CNN
regressor,initialized.
which copies
IoU
curves
obtained
by
the
CNN
regressor,
which
copies
the the mixed C
parameters from a pre-trained CNN classifier, on the aesthetic
parameters
from
a
pre-trained
CNN
classifier,
on
the
aesthetic
datasets.
9 complete te
datasets.
stage).

method
obtains a better cropping region
after several stages of
0.88
0.88
P1
P1
the same s
regression.
Note
that
the
above
cropping
region initialization
P2
P2
number of
0.86
0.86
P3
strategy P3
is very challenging, where the proposed CCR method
tries
to regress the bounding box for
the image content outside in Fig. 3. I
0.84
0.84
the cropping region obtained from the previous stage. The regressor a
0.82
0.82
proposed
CCR method gradually crops
images from inside to by using th
outside to overcome the boundary cutting problem. However, aesthetic da
0.8
0.8
results are
0
10
20
30
0 strategy
10 is not20optimal.
30
the above
cropping
region
initialization
#Stages
#Stages
As shown in Fig. 8 (c), the proposed CCR method obtains the obtained by
(a)
(b)
highest average
IoU values when it crops images
from outside worse than
Fig. 11: The IoU curves obtained by the proposed method on Fig. 8). Th
to inside
(i.e.,IoU
thecurves
originalobtained
image isby
used
the initialmethod
cropping
Fig.
11: The
theasproposed
on
the
three image cropping datasets with two different feature
region).
the
three image cropping datasets with two different feature training sam
extraction
strategies. (a) The IoU
curves
obtained
by the
Using different
models
to extract
extraction
strategies. pre-trained
(a) The IoUCNN
curves
obtained
by the affects the
proposed
method
with
the
cropping-indexed
CNN
features.
the cropping-indexed
CNN
features. To show
effecproposed
method with the
cropping-indexed
CNNthefeatures.
(b)
The IoU
curves
obtained by the proposed
method by
with B. Compar
tiveness
of the
cropping-indexed
features extracted
(b)
The IoU
curves
obtained by CNN
the proposed
method with
the
mixed
CNN features
(i.e.,
the features
extracteddatasets,
from the In this
the
CNN
model,
which
is
trained
on
the
aesthetic
the mixed CNN features (i.e., the features extracted from the
complete
test images
and the cropped
regions obtained[8])
in each
we use test
another
pre-trained
CNN model
to proposed m
complete
images
and the cropped
regions(AlexNet
obtained in each
stage).
extract
cropping-indexed
CNN
features
for
image
cropping
several stat
stage).
as a comparison. AlexNet is a pre-trained deep CNN model comparison
on the ImageNet dataset [40] for image classification and it cropping m
the
same5 structure
as that given
in
Fig. 3 except layers.
that the
includes
andonly
2 fully-connected
Fig.
8). Theconvolutional
main reasonlayers
is that
a limited number of three categ
number
of the output
nodes
in
the last
layer
issecond
4 instead
of 2
We
empirically
choose
the
output
from
the
training samples are used to train the CNN regressor, fullywhich Section II)
in
Fig. 3. Inlayer
the first
experiment,
thecropping-indexed
parameters of theCNN
CNN method [41
connected
of
AlexNet
as
the
affects the performance of the CNN regressor.
regressor
are second
randomly
initialized, while
they
are initialized
features. The
fully-connected
layer of
AlexNet
is the Both metho
by
using the parameters
a CNN
classifier
pre-trained
on the the extensi
best-performing
layer forofimage
cropping
in our
experiments.
B. Comparison with state-of-the-art methods
aesthetic
datasets in
the second
experiment.
experimental
The experimental
results
are shown
in Fig. 8 The
(d). We
can see achieve bet
In this
subsection,
we
the method [4]
results
shownIoU
in values
Fig.evaluate
10.
As the
can
be
the of
results
that
theare
average
obtained
by performance
the seen,
proposed
CCR
proposed
method
for
image
cropping
and
compare
it
with
method using
CNNregressor
model pre-trained
the aestheticare photograph
obtained
by thetheCNN
in the twoonexperiments
several
state-of-the-art
on proposed
the cropping
datasets.
The
datasets
arethose
higherobtained
thanmethods
those
by theCCR
proposed
CCR
worse
than
by obtained
the
method
(see version of
comparison
is
performed
using
four
state-of-the-art
image
method
using
the
AlexNet
CNN
model.
The
reason
is
that
the
Fig. 8). The main reason is that only a limited number of paper. The
cropping
methods,
which
aretothe
the
representative
methods
from since its so
CNN model
pre-trained
on
aesthetic
datasets
can extract
training
samples
are
used
train
the CNN
regressor,
which
three
categories
of
image
cropping
methods
(as
discussed
in
more
effective
aesthetic
features,
which
are
beneficial
to
visual
For the h
affects the performance of the CNN regressor.
aestheticII).
enhancement.
Section
The first two methods are the attention-based model on
a native
CNN
regressor with
different
parameter professiona
method
[42]
and the
aesthetic-based
method
[27], respectively.
B. Using
Comparison
with
state-of-the-art
methods
initialization
strategies.
effectiveness
of the λ = 0.800.
Both
methods are
extendedTobyshow
Yan the
et al.
[29]. For fairness,
Inextension
this CCR
subsection,
we also
evaluate
the
performance
of the
proposed
method,[29]
we
report
themethods
experimental
results
the
version
of
the
two
are used
to al photogra
proposed
method
for
image
cropping
and
compare
it
with similar to t
obtained
by
using
a
CNN
regressor
as
a
comparison.
Two
achieve better results. The third method is the change-based
several
state-of-the-art
methods
onnative
the cropping
datasets.
The IoU values
experiments
are performed
by the
CNN
regressor
with
method
[4], which
learns
the
experience
from
professional
comparison
is
performed
using
four
state-of-the-art
image
two
different
parameter
initialization
strategies
to
directly
photographers by using hand-crafted features. The improved on the thre
cropping
methods,
which aremethod
the
representative
methods
from
regress of
the
4-tuple
coordinate
vector.
The CNN
version
the
change-based
[29]
is alsoregressor
used in has
this respectively
three
categories
of
image
cropping
methods
(as
discussed
paper. The method in [23] is not evaluated for comparisonin
Section
II). The
firstis not
two available
methods toare
attention-based
since
its source
code
thethe
public.
method [41] and the aesthetic-based method [26], respectively.
Both methods are extended by Yan et al. [28]. For fairness,
the extension version [28] of the two methods are used to
achieve better results. The third method is the change-based
IoU

IoU

IoU

parameters therefore become smaller in the latter stages.
Boundary cutting problem. If the original initial cropping
region is scaled down to 50% or 25% of its original size, then
the cropping boundaries may pass through an object, which
causes an unpleasant visual effect. However, the proposed
method can alleviate this problem after several stages of
regression. As shown in Fig. 9, the boundaries of the tree
are cut in the beginning stage due to the small size of the
original initial cropping region. However, the proposed CCR
method obtains a better cropping region after several stages of
regression. Note that the above cropping region initialization
strategy is very challenging, where the proposed CCR method
tries to regress the bounding box for the image content outside
the cropping region obtained from the previous stage. The
proposed CCR method gradually crops images from inside to
IEEE TRANSACTIONS ON MULTIMEDIA
outside
to overcome the boundary cutting problem. However,
the above cropping region initialization strategy is not optimal.
As0.4shown in Fig. 8 (c), the proposed
0.45 CCR method obtains the
Train
highest average IoU values when it crops images from outside
Test
to 0.3
inside (i.e., the original image is 0.4
used as the initial cropping
region).
0.35
0.2
Using different pre-trained CNN models to extract
0.3
the0.1 cropping-indexed CNN features.
To show the effecTrain
tiveness of the cropping-indexed
CNN
features
extracted by
Test
0.25
0
0 the50aesthetic
100
150
200
0
50
100
150
the CNN
model,
which
is trained
on
datasets,
#Epochs
#Epochs
we use another pre-trained CNN model (AlexNet [9]) to
(a)
(b)
extract cropping-indexed CNN features for image cropping
Fig. 10: The IoU curves obtained by a CNN regressor with
as a comparison. AlexNet is a pre-trained deep CNN model
different parameter initialization strategies on the training
on the ImageNet dataset [41] for image classification and it
and test datasets. (a) The IoU curves obtained by the CNN
includes 5 convolutional layers and 2 fully-connected layers.
regressor whose parameters are randomly initialized. (b) The
We empirically choose the output from the second fullyIoU curves obtained by the CNN regressor, which copies the
connected layer of AlexNet as the cropping-indexed CNN
parameters from a pre-trained CNN classifier, on the aesthetic
features. The second fully-connected layer of AlexNet is the
datasets.
best-performing layer for image cropping in our experiments.
The experimental results are shown in Fig. 8 (d). We can see
that the average IoU values obtained by the proposed CCR
method obtains a better cropping region after several stages of
method using the CNN model pre-trained on the aesthetic
regression. Note that the above cropping region initialization
datasets are higher than those obtained by the proposed CCR
strategy is very challenging, where the proposed CCR method
method using the AlexNet CNN model. The reason is that the
tries to regress the bounding box for the image content outside
CNN model pre-trained on the aesthetic datasets can extract
the cropping region obtained from the previous stage. The
more effective aesthetic features, which are beneficial to visual
proposed CCR method gradually crops images from inside to
aesthetic enhancement.
outside to overcome the boundary cutting problem. However,
Using a native CNN regressor with different parameter
the above cropping region initialization strategy is not optimal.
initialization strategies. To show the effectiveness of the
As shown in Fig. 8 (c), the proposed CCR method obtains the
proposed CCR method, we also report the experimental results
highest average IoU values when it crops images from outside
obtained by using a CNN regressor as a comparison. Two
to inside (i.e., the original image is used as the initial cropping
experiments are performed by the native CNN regressor with
region).
two different parameter initialization strategies to directly
Using different pre-trained CNN models to extract
regress the 4-tuple coordinate vector. The CNN regressor has
the cropping-indexed CNN features. To show the effecthe same structure as that given in Fig. 3 except that the
tiveness of the cropping-indexed CNN features extracted by
number of the output nodes in the last layer is 4 instead of 2
the CNN model, which is trained on the aesthetic datasets,
in Fig. 3. In the first experiment, the parameters of the CNN
we use another pre-trained CNN model (AlexNet [8]) to
regressor are randomly initialized, while they are initialized
extract cropping-indexed CNN features for image cropping
by using the parameters of a CNN classifier pre-trained on the
as a comparison. AlexNet is a pre-trained deep CNN model
aesthetic datasets in the second experiment. The experimental
on the ImageNet dataset [40] for image classification and it
results are shown in Fig. 10. As can be seen, the results
includes 5 convolutional layers and 2 fully-connected layers.
obtained by the CNN regressor in the two experiments are
We empirically choose the output from the second fullyworse than those obtained by the proposed CCR method (see
connected layer of AlexNet as the cropping-indexed CNN
features. The second fully-connected layer of AlexNet is the
best-performing layer for image cropping in our experiments.
The experimental results are shown in Fig. 8 (d). We can see

9

IEEE TRANSACTIONS ON MULTIMEDIA

IoU

IEEE TRANSACTIONS ON MULTIMEDIA

IEEE
IEEETRANSACTIONS
TRANSACTIONSON
ON MULTIMEDIA
MULTIMEDIA

(a)

(b)

1010

(c)

(d)

(e)

(f)

(g)

Fig. 12:
12: Qualitative
Qualitative examples
examples showing
showing the
the cropped
cropped images
images obtained
Fig.
obtained by
by the
the five
five competing
competing methods.
methods. (a)
(a) The
The original
originalImages.
Images.
(b)
The
cropped
images
annotated
by
the
first
professional
photographer.
(c)-(g)
The
cropping
results
obtained
by
(b) The cropped images annotated by the first professional photographer. (c)-(g) The cropping results obtained bythe
theattentionattentionbased [42],
[41], the
the aesthetic-based
aesthetic-based [27],
[26], the
the change-based
change-based (2013)
(2013) [4],
based
[4], the
the change-based
change-based (2015)
(2015) [28],
[29], and
and the
the proposed
proposedmethods.
methods.
TABLE II: The IoU (and BDE) results obtained by the four competing state-of-the-arts methods and the proposed method for
image cropping.
The results
obtained
by the
four competing
methods
are from [28].In each stage, the pre-trained CNN
TABLE
I: The test
accuracy
obtained
by other
the pre-trained
region
for comparison.
CNN classifier on the original cropping dataset and the three model is respectively applied on the complete test image and
Methods
Photographer1 Photographer2 Photographer3
cropping datasets cropped by the proposed CCR method.
the cropped region obtained from the previous stage, to extract
Attention-based [41]

Datasets
Accuracy
Aesthetic-based [26]
original
0.2512
Change-based
Photographer1
0.4370 (2013) [4]
Change-based
Photographer2
0.4141 (2015) [28]
The proposed
Photographer3
0.3815method

0.203
0.396
0.749
0.797
0.850

(0.254)
0.178 (0.200)
0.199 (0.259)
the features.
The features
obtained from both the complete test
(0.178)
0.394 (0.178)
0.386 (0.183)
image
and
the
cropped
region
are concatenated as a vector,
(0.067)
0.729 (0.072)
0.732 (0.072)
which
is
used
as
the
mixed
CNN
(0.053)
0.786 (0.057)
0.772 (0.059) features. The experimental
(0.032)
0.82811.
(0.035)
results0.837
are (0.033)
shown in Fig.
As can be seen, the IoU values

TABLE I: The test accuracy obtained by the pre-trained
CNN
on the original
cropping
dataset
and the CCR
three
For classifier
the hyperparameter
setting
used for
training
cropping
croppeddataset
by the[29]
proposed
CCR method.
model
on datasets
the cropping
annotated
by the first
professional photographer,
S = 4, M = 20 and
Datasets we setAccuracy
originalannotated 0.2512
λ = 0.800. The images
by the other two profes0.4370
sional photographersPhotographer1
are used for test.
This evaluation strategy
Photographer2
0.4141
is similar to that usedPhotographer3
in [4], [29]. As
shown
in Fig. 11(a), the
0.3815
IoU values obtained by the proposed method at the 30th stage
on the three cropping datasets are 0.850, 0.837 and 0.828,
respectively. However, we note that the proposed CCR can
also achieve promising results even at the 10th stage, where
also achieve promising results even at the 10th stage, where
the IoU values obtained by the proposed CCR method on the
the IoU values obtained by the proposed CCR method on the
three datasets are respectively 0.839, 0.834 and 0.823.
three datasets are respectively 0.839, 0.834 and 0.823.
To show the effectiveness of the proposed cropping-indexed
To show
the effectiveness
proposed cropping-indexed
CNN
features,
we evaluate of
thethe
performance
of the proposed
CNN
features,
we
evaluate
the
performance
the proposed
method with the mixed CNN features that areofsimultaneously
method
with
the
mixed
CNN
features
that
are
simultaneously
extracted from both a complete test image and the cropped
extracted from both a complete test image and the cropped

obtained by the proposed method with the mixed features at
the 30th stage on the three cropping datasets are respectively
0.838, 0.834
and 0.823, In
which
lower
those obtained
region
for comparison.
eachare
stage,
thethan
pre-trained
CNN
by
the
proposed
method
with
only
the
cropping-indexed
model is respectively applied on the complete test image CNN
and
features.
Theregion
reasonobtained
is that the
cropping-indexed
CNN
the
cropped
from
the previous stage,
to features
extract
can features.
provide The
aesthetic
discriminative
Moreover,
the
features
obtained frominformation.
both the complete
test
the
proposed
primitive
regressor
can
select
the
features,
which
image and the cropped region are concatenated as a vector,
are beneficial
cropping
visual aesthetic
enhancewhich
is used to
as image
the mixed
CNNfor
features.
The experimental
ment.
In
contrast,
the
mixed
CNN
features
include
many
results are shown in Fig. 11. As can be seen, the IoU values
redundant
features,
which
will
lead
to
high
computational
obtained by the proposed method with the mixed features at
time30th
andstage
the overfitting
problem.
More
specifically,
effective
the
on the three
cropping
datasets
are respectively
features
are
hard
to
be
selected
from
the
mixed
CNN
features
0.838, 0.834 and 0.823, which are lower than those obtained
the proposed
proposedmethod
primitive
that the length
by the
withregressor
only the because
cropping-indexed
CNN
of the mixed
CNNis that
feature
vector is higherCNN
thanfeatures
that of
features.
The reason
the cropping-indexed
the cropping-indexed
feature vector.
Thus, Moreover,
the mixed
can
provide aesthetic CNN
discriminative
information.
CNN
featuresprimitive
are lessregressor
effective
thethecropping-indexed
the
proposed
canthan
select
features, which
CNN
featurestofor
cropping
region
regression.
Additionally,
are
beneficial
image
cropping
for visual
aesthetic
enhancement.
In contrast,
mixed CNN
include
many
the pre-trained
CNNtheclassifier
is usedfeatures
to classify
the images
from the original cropping dataset and the cropped images

IEEE TRANSACTIONS ON MULTIMEDIA

11

TABLE II: The IoU (and BDE) results obtained by the four competing state-of-the-arts methods and the proposed method for
image cropping. The results obtained by the other four competing methods are from [29].
Methods

Photographer1

Photographer2

Photographer3

Attention-based [42]
Aesthetic-based [27]
Change-based (2013) [4]
Change-based (2015) [29]
The proposed method

0.203
0.396
0.749
0.797
0.850

0.178
0.394
0.729
0.786
0.837

0.199
0.386
0.732
0.772
0.828

(0.254)
(0.178)
(0.067)
(0.053)
(0.032)

obtained by the proposed CCR method, respectively. The
classify accuracies are shown in Table 1. As can be seen,
the obtained classification accuracies on the cropped images
are higher than those on the original images about 13%-18%,
which shows the effectiveness of the proposed image cropping
method.
The results obtained by the proposed method are compared
with those obtained by the other four competing methods.
Table II lists the comparison results. Some examples of the
cropping regions annotated by the first professional photographer and the cropping regions obtained by the five competing
methods are also shown in Fig. 12. As we can see from
Table II and Fig. 12, the proposed CCR method clearly outperforms the other four competing methods, and the cropping
regions obtained by the proposed CCR method are closer
to those annotated by the professional photographers than
the other competing methods. The change-based method [4]
and its improved version [29] are similar to the proposed
CCR method because they also learn the experience from
professional photographers. However, due to the fact that the
hand-crafted features used in the change-based method and
its improved version are often limited, both methods obtain
worse performance than the proposed method. Among the five
competing methods, the attention-based method [42] obtains
the worst IoU and BDE results on all the three cropping
datasets since it only focuses on the salient objects in an
image while it ignores the aesthetic of a cropped image. The
aesthetic-based method [27] achieves better results than the
attention-based method [42], but it performs worse than the
change-based method. This is because that the aesthetic-based
method focuses more on high-quality local cropping regions,
while it may ignore the object regions in an image. The
proposed method and the change-based methods can overcome
this drawback by learning the knowledge of the professional
photographers. Thus, they can obtain better results than the
aesthetic-based method.
To analyze the failure case of the proposed method, we
also give some failure examples (see Fig. 13) obtained by
the proposed method. Two kinds of images may cause the
failure of the proposed method. The first kind of images are
those in which the position of an object is close to the image
boundaries. The proposed method may cut off a small part of
the object in an image if the regression results are not accurate
enough. As shown in the first failure example in Fig. 13(a), a
part of the reflection of the bird is removed by the proposed
method. The second kind of images usually have no much
texture in their backgrounds. As shown in the first failure

(0.200)
(0.178)
(0.072)
(0.057)
(0.033)

(0.259)
(0.183)
(0.072)
(0.059)
(0.035)

TABLE III: The numbers of failure cases (denoted by #Failure) obtained by the proposed CCR method in the cropping
datasets.
Dataset

Photographer1

Photographer2

Photographer3

#Failure

41

60

56

example in the figure, the background of the example is a
solid color. The proposed method may not work quite well for
the second kind of images. The total numbers of failure cases
obtained by the proposed CCR method on the three cropping
datasets are listed in Table III. As can be seen, the proposed
CCR method only fails on a small percentage of images (4.3%
- 6.3% of the whole test images).
In terms of running time, the total execution time of the
proposed method (implemented in Matlab on a 4 GHz 32GB
RAM PC) on each image is about 1.7 seconds. Most running
time is used to extract the cropping-indexed CNN features.
In contrast, the running time of the improved change-based
method (implemented in C++ on a 2.33 GHz 4GB RAM PC)
is about 11 seconds [29].
V. C ONCLUSION
In this paper, we propose a novel CCR method to perform
automatic image cropping for visual aesthetic enhancement
based on the cropping-indexed CNN features. A two-step
learning strategy is proposed for a limited number of labelled
cropping data. In addition, the proposed CCR method learns
a primitive regressor by using the gradient boosting strategy
on a group of random-ferns regressors at each stage. The
convergence speed of the proposed CCR method is much faster
than that of the cascaded regression method directly using
random-ferns regressors. On the other hand, the performance
of the proposed CCR method for image cropping also benefits
from the cropping-indexed CNN features, which are extracted
by the CNN model trained on large-scale visual aesthetic
datasets. Experimental results show that the proposed method
is quite effective and efficient for image cropping.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science
Foundation of China under Grants U1605252, 61472334 and
61571379, and by the Natural Science Foundation of Fujian
Province of China under Grant 2017J01127.

h
f
d
d
,
s
s
e
,
s
,
g

d
.
e
g
m
g
r
n
]
d
m
e
d
n
e
s
g
n
e
e
e
d
,
e
e
l
e

e
y
e
e
e
f
e
a
d
h

Dataset

Photographer1

Photographer2

#Failure
41ON MULTIMEDIA
IEEE TRANSACTIONS

60

Photographer3
56

12

[10] X. Lu, Z. Lin, H. Jin, J. Yang, and J. Z. Wang, “Rapid: Rating pictorial
aesthetics using deep learning,” in ACM MM, 2014, pp. 457–466.
[11] L. Mai, H. Jin, and F. Liu, “Composition-preserving deep photo aesthetics assessment,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
(CVPR), 2016, pp. 497–506.
[12] F. Gao, Y. Wang, P. Li, M. Tan, J. Yu, and Y. Zhu, “Deepsim: Deep
similarity for image quality assessment,” Neurocomputing (NC), vol.
257, no. 1, pp. 104 – 114, 2017.
[13] N. Murray, L. Marchesotti, and F. Perronnin, “Ava: A large-scale
database for aesthetic visual analysis,” in Proc. IEEE Conf. Comput.
Vis. Pattern Recognit. (CVPR), 2012, pp. 2408–2415.
[14] F. Gao, D. Tao, X. Gao, and X. Li, “Learning to rank for blind
image quality assessment,” IEEE Transactions on Neural Networks and
Learning Systems (TNNLS), vol. 26, no. 10, pp. 2275–2290, 2015.
[15] A. K. Moorthy and A. C. Bovik, “Blind image quality assessment: From
natural scene statistics to perceptual quality,” IEEE Trans. on Image
Processing (TIP), vol. 20, no. 12, pp. 3350–3364, 2011.
[16] B. Cheng, B. Ni, S. Yan, and Q. Tian, “Learning to photograph,” in Int.
Conf. on Multimedia (ICMM), 2010, pp. 291–300.
[17] K. Koffka, Principles of Gestalt psychology. Harcourt, Brace New
York, 1935.
[18] P. Welinder and P. Perona, “Cascaded pose regression,” in Proc. IEEE
Conf. Comput. Vis. Pattern Recognit. (CVPR), 2010, pp. 1078–1085.
[19] J. H. Friedman, “Greedy function approximation: A gradient boosting
machine,” Annals of Statistics, vol. 29, pp. 1189–1232, 2001.
[20] F. Perazzi, Y. Pritch, and A. Hornung, “Saliency filters: Contrast based
filtering for salient region detection,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit. (CVPR), 2012, pp. 733–740.
[21] G. Guo, H. Wang, W. L. Zhao, Y. Yan, and X. Li, “Object discovery
via cohesion measurement,” IEEE Trans. on Cybernetics (TCYB), vol. 1,
no. 99, pp. 1–14, 2017.
[22] L. Marchesotti, C. Cifarelli, and G. Csurka, “A framework for visual
saliency detection with applications to image thumbnailing,” in Proc.
IEEE Int. Conf. Comput. Vis. (ICCV), 2009, pp. 2232–2239.
[23] C. Fang, Z. Lin, R. Mech, and X. Shen, “Automatic image cropping
using visual composition, boundary simplicity and content preservation
models,” in ACM MM, 2014, pp. 1105–1108.
(a)
(b)
[24] G. Ciocca, C. Cusano, F. Gasparini, and R. Schettini, “Self-adaptive
cropping for small displays,” in International Conference on
Fig. 13: Failure examples obtained by the proposed method.image
Consumer Electronics, 2007, pp. 1–2.
From top
each row
shows
cropping
results[25]
an-T. N. Vikram, M. Tscherepanow, and B. Wrede, “A saliency map based
Fig. to
13: bottom,
Failure examples
obtained
by the
the proposed
method.
sampling an image into random rectangular regions of interest,”
From
top
to
bottom,
each
row
shows
the
cropping
results
annotated by the first professional photographer and the croppingon
Pattern Recognition (PR), vol. 45, no. 9, pp. 3114–3124, 2012.
notated by the first professional photographer and the cropping [26] A. Laurentini and A. Bottino, “Computer analysis of face beauty: A
resultsresults
obtained
by the proposed method, respectively.
obtained by the proposed method, respectively.
survey,” Computer Vision and Image Understanding (CVIU), vol. 125,
pp. 184 – 199, 2014.
[27] M. Nishiyama, T. Okabe, Y. Sato, and I. Sato, “Sensation-based photo
cropping,” in ACM MM, 2009, pp. 669–672.
EFERENCES
solid color. The proposed Rmethod
may not work quite well[28]
forL. Zhang, M. Song, Q. Zhao, X. Liu, J. Bu, and C. Chen, “Probabilistic
graphlet transfer for photo cropping,” IEEE Transactions on Image
the second
ofDatta,
images.
The total
of J.failure
[1] D.kind
Joshi, R.
E. Fedorovskaya,
Q.-T.numbers
Luong, J. Wang,
Li, and casesProcessing, vol. 22, pp. 802–815, 2013.
J. Luo, “Aesthetics and emotions in images,” IEEE Signal Processing [29] J. Yan, S. Lin, S. Kang, and X. Tang, “Change-based image cropping
obtained by
the proposed
CCR2011.
method on the three croppingwith exclusion and compositional features,” International Journal of
Magazine,
vol. 28, pp. 94–115,
[2]
R.
Datta,
D.
Joshi,
J.
Li,
and
J.
Z. Wang,
“Studying
in
Computer Vision (IJCV), vol. 114, pp. 1–14, 2015.
datasets are
listed in Table III. As
can be
seen,aesthetics
the proposed
photographic images using a computational approach,” in Proc. Eur. [30] “Robust non-convex least squares loss function for regression with
CCR method
only
on a small
of images (4.3%outliers,” Knowledge-Based Systems, vol. 71, no. 1, pp. 290 – 302, 2014.
Comput.
Vis. fails
Conf. (ECCV),
2006, pp.percentage
288–301.
[3] L. Marchesotti, F. Perronnin, D. Larlus, and G. Csurka, “Assessing the
[31] J. Yu, X. Yang, F. Gao, and D. Tao, “Deep multimodal distance metric
- 6.3% of aesthetic
the whole
test
images).
quality of
photographs
using generic image descriptors,” in
learning using click constraints for image ranking,” IEEE Transactions
Int. Conf. time,
Comput. the
Vis. (ICCV),
pp. 1784–1791.
In termsProc.ofIEEE
running
total2011,
execution
time of theon Cybernetics (TCYB), vol. 1, no. 99, pp. 1–11, 2017.
[4] J. Yan, S. Lin, S. B. Kang, and X. Tang, “Learning the change for
[32] W. Luo, X. Wang, and X. Tang, “Content-based photo quality assessproposed method
(implemented
inIEEE
Matlab
on a Vis.
4 GHz
automatic image
cropping,” in Proc.
Conf. Comput.
Pattern 32GBment,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2011, pp. 2206–
Recognit. (CVPR), 2013, pp. 971–978.
2213.
RAM PC)
each
image
is about
seconds.
Most
running
[5] L.on
Zhang,
M. Song,
Y. Yang,
Q. Zhao, 1.7
C. Zhao,
and N. Sebe,
“Weakly
[33] G. E. Dahl, T. N. Sainath, and G. E. Hinton, “Improving deep neural
supervised
photo cropping,”
Transactions on Multimedia
(TMM),
time is used
to extract
the IEEE
cropping-indexed
CNN
features.networks for LVCSR using rectified linear units and dropout,” in
vol. 16, no. 1, pp. 94–107, 2014.
ICASSP, 2013, pp. 8609–8613.
In contrast,
theT. running
time
improved
change-based
[6] W. Yin,
Mei, C. W. Chen,
and of
S. Li,the
“Socialized
mobile photography:
[34] K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in
Learning to photograph with social context via mobile devices,” IEEE
deep convolutional networks for visual recognition,” IEEE Trans. Pattern
method (implemented
in C++
on avol.2.33
4GB
Transactions on Multimedia
(TMM),
16, pp.GHz
184–200,
2014. RAM PC)Anal. Mach. Intell. (TPAMI), vol. 37, no. 9, pp. 1904–1916, 2015.
A. Jahanian,
S. V.[28].
N. Vishwanathan, and J. P. Allebach, “Learning visual
[35] R. Collobert, K. Kavukcuoglu, and C. Farabet, “Torch7: A matlab-like
is about[7]11
seconds
balance from large-scale datasets of aesthetically highly rated images,”
environment for machine learning,” in Workshop on Proc. Adv. Neural
in IS&T/SPIE Electronic Imaging, 2015, pp. 93 940Y–93 940Y–9.
Inf. Process. Syst. (WNIPS), 2011, pp. 1–6.
[8] X. Tang, W. Luo, and X. Wang, “Content-based photo quality assess[36] X. Cao, Y. Wei, F. Wen, and J. Sun, “Face alignment by explicit shape
V.
C
ONCLUSION
ment,” IEEE Transactions on Multimedia (TMM), vol. 15, no. 8, pp.
regression,” International Journal of Computer Vision (IJCV), vol. 107,
1930–1943, 2013.
pp. 177–190, 2014.
In this
we I.propose
a novel
CCR
method
to perform
[9] paper,
A. Krizhevsky,
Sutskever, and
G. E. Hinton,
“Imagenet
classification
[37] Y. Sun, X. Wang, and X. Tang, “Deep convolutional network cascade
deep convolutional neural networks,” in Proc. Adv. Neural Inf.
facial point detection,” in Proc. IEEE Conf. Comput. Vis. Pattern
automatic with
image
cropping for visual aesthetic enhancementfor
Recognit. (CVPR), 2013, pp. 3476–3483.
Process. Syst. (NIPS), 2012, pp. 1097–1105.

based on the cropping-indexed CNN features. A two-step
learning strategy is proposed for a limited number of labelled
cropping data. In addition, the proposed CCR method learns

IEEE TRANSACTIONS ON MULTIMEDIA

[38] L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone, Classification
and Regression Trees. New York: Chapman & Hall, 1984.
[39] M. Ozuysal, M. Calonder, V. Lepetit, and P. Fua, “Fast keypoint
recognition using random ferns,” IEEE Trans. Pattern Anal. Mach. Intell.
(TPAMI), vol. 32, pp. 448–461, 2010.
[40] N. Duffy and D. Helmbold, “Boosting methods for regression,” Machine
Learning, vol. 47, pp. 153–200, 2002.
[41] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,
Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and
L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,”
International Journal of Computer Vision (IJCV), vol. 115, pp. 211–
252, 2015.
[42] F. Stentiford, “Attention based auto image cropping,” ICVS Workshop
on Computational Attention & Application, 2007.

Guanjun Guo is currently pursuing the Ph.D. degree with the Fujian Key Laboratory of Sensing
and Computing for Smart City, and the School
of Information Science and Engineering, Xiamen
University, Xiamen, China. His current research interests include computer vision, machine learning.

Hanzi Wang (SM’10) is currently a Distinguished
Professor of Minjiang Scholars in Fujian province
and a Founding Director of the Center for Pattern Analysis and Machine Intelligence (CPAMI) at
XMU. He was an Adjunct Professor (2010-2012)
and a Senior Research Fellow (2008-2010) at the
University of Adelaide, Australia; an Assistant Research Scientist (2007-2008) and a Postdoctoral Fellow (2006-2007) at the Johns Hopkins University;
and a Research Fellow at Monash University, Australia (2004-2006). He received his Ph.D degree in
Computer Vision from Monash University where he was awarded the Douglas
Lampard Electrical Engineering Research Prize and Medal for the best PhD
thesis in the department. His research interests are concentrated on computer
vision and pattern recognition including visual tracking, robust statistics,
object detection, video segmentation, model fitting, etc. He has published more
than 100 papers in major international journals and conferences including the
IEEE T-PAMI, IJCV, ICCV, CVPR, ECCV, NIPS, MICCAI, etc. He is a senior
member of the IEEE. He was an Associate Editor for IEEE Transactions on
Circuits and Systems for Video Technology (from 2010 to 2015) and a Guest
Editor of Pattern Recognition Letters (September 2009). He was the General
Chair for ICIMCS2014, Program Chair for CVRS2012, Publicity Chair for
IEEE NAS2012, and Area Chair for ACCV2016, DICTA2010. He also serves
on the program committee (PC) of ICCV, ECCV, CVPR, ACCV, PAKDD,
ICIG, ADMA, CISP, etc, and he serves on the reviewer panel for more than
40 journals and conferences.

13

Chunhua Shen is a Professor at School of Computer
Science, University of Adelaide. He is a Project
Leader and Chief Investigator at the Australian Research Council Centre of Excellence for Robotic
Vision (ACRV), for which he leads the project
on machine learning for robotic vision. Before he
moved to Adelaide as a Senior Lecturer, he was with
the computer vision program at NICTA (National
ICT Australia), Canberra Research Laboratory for
about six years. His research interests are in the intersection of computer vision and statistical machine
learning. Recent work has been on real-time object detection, large-scale
image retrieval and classification, and scalable nonlinear optimization. He
studied at Nanjing University, at Australian National University, and received
his PhD degree from the University of Adelaide. From 2012 to 2016, he holds
an Australian Research Council Future Fellowship. He is serving as Associate
Editor of IEEE Transactions on Neural Networks and Learning Systems.

Yan Yan is currently an associate professor in the
School of Information Science and Technology at
Xiamen University, China. He received the Ph.D.
degree in Information and Communication Engineering from Tsinghua University, China, in 2009. He
worked at Nokia Japan R&D center as a research
engineer (2009-2010) and Panasonic Singapore Lab
as a project leader (2011). He has published around
40 papers in the international journals and conferences including the IEEE T-IP, T-Cyber, T-ITS, PR,
KBS, Neurocomputing, ICCV, ECCV, ACM MM,
ICPR, ICIP, etc. His research interests include computer vision and pattern
recognition.

Hong-Yuan Mark Liao (F’13) received the PhD
degree in electrical engineering from Northwestern
University, Evanston, IL, in 1990. In 1991, he joined
the Institute of Information Science, Academia
Sinica, Taipei, Taiwan, where he is currently a
distinguished research fellow. He was in the fields
of multimedia signal processing, image processing,
computer vision, pattern recognition, video forensics, and multimedia protection for more than 25
years. He received the Young Investigators Award
from Academia Sinica in 1998, the Distinguished
Research Award from the National Science Council of Taiwan, in 2003, 2010,
and 2013, respectively, the National Invention Award of Taiwan in 2004,
the Distinguished Scholar Research Project Award from the National Science
Council of Taiwan in 2008, and the Academia Sinica Investigator Award in
2010. His professional activities include the cochair of the 2004 International
Conference on Multimedia and Exposition (ICME), the Technical cochair of
the 2007 ICME, the General cochair of the 17th International Conference
on Multimedia Modeling, the President of the Image Processing and Pattern
Recognition Society of Taiwan (2006C2008), an Editorial Board member
of the IEEE Signal Processing Magazine (2010C2013), and an associate
editor of the IEEE Transactions On Image Processing (2009C2013), the IEEE
Transactions On Information Forensics And Security (2009C2012), and the
IEEE Transactions On Multimedia (1998C2001). He also serves as the IEEE
Signal Processing Society Region 10 Director (Asia-Pacific Region). He is a
fellow of the IEEE.

