Image aesthetics assessment has been challenging due to its subjective nature. Inspired by the scientific advances in the human visual perception and neuroaesthetics, we design Brain-Inspired Deep Networks (BDN) for this task. BDN first learns attributes through the parallel supervised pathways, on a variety of selected feature dimensions. A high-level synthesis network is trained to associate and transform those attributes into the overall aesthetics rating. We then extend BDN to predicting the distribution of human ratings, since aesthetics ratings are often subjective. Another highlight is our first-of-its-kind study of label-preserving transformations in the context of aesthetics assessment, which leads to an effective data augmentation approach. Experimental results on the AVA dataset show that our biological inspired and task-specific BDN model gains significantly performance improvement, compared to other state-of-the-art models with the same or higher parameter capacity.
@cite_35: Effective visual features are essential for computational aesthetic quality rating systems. Existing methods used machine learning and statistical modeling techniques on handcrafted features or generic image descriptors. A recently-published large-scale dataset, the AVA dataset, has further empowered machine learning based approaches. We present the RAPID (RAting PIctorial aesthetics using Deep learning) system, which adopts a novel deep neural network approach to enable automatic feature learning. The central idea is to incorporate heterogeneous inputs generated from the image, which include a global view and a local view, and to unify the feature learning and classifier training using a double-column deep convolutional neural network. In addition, we utilize the style attributes of images to help improve the aesthetic quality categorization accuracy. Experimental results show that our approach significantly outperforms the state of the art on the AVA dataset.
@cite_33: In this tutorial, we define and discuss key aspects of the problem of computational inference of aesthetics and emotion from images. We begin with a background discussion on philosophy, photography, paintings, visual arts, and psychology. This is followed by introduction of a set of key computational problems that the research community has been striving to solve and the computational framework required for solving them. We also describe data sets available for performing assessment and outline several real-world applications where research in this domain can be employed. A significant number of papers that have attempted to solve problems in aesthetics and emotion inference are surveyed in this tutorial. We also discuss future directions that researchers can pursue and make a strong case for seriously attempting to solve problems in this research domain.
@cite_16: Visual quality (VisQ) representation is a fundamental step in the learning of a VisQ prediction model for photos. It not only reflects how we understand VisQ but also determines the label type. Existing studies apply a scalar value (i.e., a categorical label or a score) to represent VisQ. As VisQ is a subjective property, only a scalar value is insufficient to represent human's perceived VisQ of a photo. This study represents VisQ by a distribution on pre-defined ordinal basic ratings in order to capture the subjectivity of VisQ better. When using the new representation, the label type is structural instead of scalar. Conventional learning algorithms cannot be directly applied in model learning. Meanwhile, for many photos, the numbers of users involved in the evaluation are limited, making some labels unreliable. In this study, a new algorithm called support vector distribution regression (SVDR) is presented to deal with the structural output learning. Two independent learning strategies (reliability-sensitive learning and label refinement) are proposed to alleviate the difficulty of insufficient involved users for rating. Combining SVDR with the two learning strategies, two separate structural-output regression algorithms (i.e., reliability-sensitive SVDR and label refinement-based SVDR) are produced. Experimental results demonstrate the effectiveness of our introduced learning strategies and learning algorithms.
@cite_15: Current research of aesthetic assessment for images assumes one aesthetic score or one aesthetic label for an image, ignoring the relations of multiple aesthetic-related attributes. However, most images can be described by multiple aesthetic attributes simultaneously. Therefore, in this paper, we propose multiple aesthetic attribute prediction and classification by modeling relations among aesthetic attributes through Bayesian Networks (BN). In order to realize continuous aesthetic attribute prediction, each aesthetic attribute is represented by a three-node BN, including the discretized aesthetic attribute label, the predicted aesthetic attribute score, and the measurement of the aesthetic attribute score. In addition, the relations among multiple aesthetic attributes are modeled by another discrete BN, whose structure and conditional probabilities are learned from the training data. The attribute measurements are obtained by an existing image-driven regression method. With the learned BN, we infer the true discrete label and continuous score for each attribute by combining the relations among attributes with the previously obtained measurements. Experiments on the Memorability dataset show the superiority of our proposed approach to current image-driven methods for both multiple continuous aesthetic attribute score prediction and multiple discrete aesthetic attribute label classification, indicating the effectiveness of the captured relations for aesthetic quality assessment.
@cite_34: We propose a principled method for designing high level features forphoto quality assessment. Our resulting system can classify between high quality professional photos and low quality snapshots. Instead of using the bag of low-level features approach, we first determine the perceptual factors that distinguish between professional photos and snapshots. Then, we design high level semantic features to measure the perceptual differences. We test our features on a large and diverse dataset and our system is able to achieve a classification rate of 72 on this difficult task. Since our system is able to achieve a precision of over 90 in low recall scenarios, we show excellent results in a web image search application.
Most current work treat aesthetics assessment as a conventional classification problem: the user ratings of each photo are transformed into a ordinal scalar rating (by averaging, etc.), which is taken as the label of the photo. For example, RAPID @cite_35 simply divided all samples as aesthetic or unaesthetic, and trained a binary classification model. However, it is common for different users to rate visual subjects inconsistently or even oppositely due to the subjective problem nature @cite_34 . Since human aesthetic assessment depends on multiple dimensions such as composition, colorfulness, or even emotion @cite_33 , it is difficult for individuals to reliably convert their experiences to a single rating, resulting in noisy estimates of real aesthetic responses. In @cite_16 , Wu et.al. first proposed to represent each photo's rating as a distribution vector over basic ratings, constituting a structural regression problem. Gao et.al. @cite_15 formulated the aesthetic assessment as a multi-label task, where multiple aesthetic attributes were predicted jointly via bayesian networks.