{'fraction_non_alphanumeric': 0.021963643160895368, 'fraction_numerical': 0.008458339174727792, 'mean_word_length': 4.989364679541002, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Sample-specific repetitive learning for photo aesthetic assessment and highlight region extraction

Aesthetic assessment is subjective, and the distribution of the aesthetic levels is imbalanced. In order to realize the auto-assessment of photo aesthetics, we focus on retraining the CNN-based aesthetic assessment model by dropping out the unavailable samples in the middle levels from the training data set repetitively to overcome the effect of imbalanced aesthetic data on classification. Further, the method of extracting aesthetics highlight region of the photo image by using the two repetitively trained models is presented. Therefore, the correlation of the extracted region with the aesthetic levels is analyzed to illustrate what aesthetics features influence the aesthetic quality of the photo. Moreover, the testing data set is from the different data source called 500px. Experimental results show that the proposed method is effective.


Introduction
In response to the growth of digital camera, more and more pictures are taken to upload the social media. Many people hope to improve aesthetic level of themselves by taking beautiful photographs. So, auto-assessment of photo aesthetics is challenging.
Researches have been investigating methods for providing automated aesthetical evaluation and classification of photographs. Aesthetic assessment is subjective. One of the main difficulties in addressing this challenge is in developing formal models of human aesthetic preference [1]. In this paper, authors stated that such models would allow computer systems to predict the aesthetic taste of a human being or adapt to the aesthetic tendencies of a human group. For making aesthetics automatic evaluation and choices, the best way to proceed is to create datasets for training the model in collaboration psychology aesthetics (PA) researchers, because computational aesthetics (CA) research typically reposts results using a success rate, while psychologists are more likely to use correlation. Closer collaboration between CA and PA can give rise to results that advance both disciplines. In [2], recent computer vision techniques used in the assessment of image aesthetic quality were reviewed. The authors discussed the possibility of manipulating the aesthetics of images through computational approaches.
The research reviewed in the paper generally aims at assessing the aesthetic quality of photos with aesthetic scores or distinguishing high-quality photos form low-quality photos, by training the photo aesthetic models based on the deep learning techniques.
However, such models can't interpret which salient image composition features and highlight regions are correlated with the photo aesthetics. Moreover, who labeled the aesthetic scores of training data set for deep learning, professional photographer or amateur, is unclear. In [3], a set of features derived from both low-and high-level analysis of photo layout were exploited to perform the aesthetic quality evaluation by a Support Vector Machine (SVM) classifier. In [4], authors designed a set of compact rulebased features based on photographic rules and aesthetic attributes, and used Deep Convolutional Neural Network (DCNN) descriptor to implicitly describe the photo quality. These approaches focused on extracting the handcrafted image features.
However, the effectiveness is limited that extracting the features is based on the researchers' understanding on the aesthetic rules. In [5], the images were divided into three categories: "scene", "object" and "texture". Each category has an associated convolutional neural network (CNN) which learns the aesthetic features for the category classification. In [6], a scene convolutional layer was designed to learn specific aesthetic features for various scenes by deep learning model. In [7], a novel photograph aesthetic classifier with a deep and wide CNN for fine-granularity aesthetical quality prediction was introduced. However, the correlation of the extracted features with the photo aesthetic assessment was not interpreted in the view of PA in such research. In [8], the percentage distributions for orientation, curvature, color and global symmetry were extracted and fed to a deep neural network under the form of only 114 inputs. Differences in extracted features between aesthetically good and poor images were analyzed and some human aesthetic preferences in static two-dimensional scenes were observed.
However, the issue whether the handcrafted features are generic for the photo aesthetic assessment is not involved. Moreover, all of the above approaches were not involved in the issue that the aesthetic rating is ambiguous and is different from person to person, which caused a highly imbalanced distribution of aesthetic ratings. Toward to tackling these issues, authors in [9] showed how to learn deep features for imbalanced data classification. Using the learned features, the classification was simply achieved by a fast cluster-wise kNN search followed by a local large margin decision. In [10], authors proposed an end-to-end CNN model which simultaneously implements aesthetic classification and understanding. A sample-specific classification method that re-weights samples' importance is implemented, and what is learned in the deep model was investigated. Ambiguous samples are given lower weights while clear samples are weighted high. However, the method to give the weight of every sample was not explicit, and the improvement for the imbalanced data classification was not salient from the experiment results. Further, the correlation of the learned deep features with the aesthetic assessment was not analyzed although deep activation map was visualized.
Motivated by the above research, we collected about 3100 photos scored aesthetically by a professional photographer who could be considered as a PA researcher. These photos were taken by the students of the photographer's class. The scores are in the range of [2,9]. The photos with score 2 or less are aesthetically poor; those with score 3 to 4 are fair; those with score 5 and 6 are good; those with 7 or more are excellent. The data set indeed exhibited a highly non-uniform distribution over score as illustrated in Fig  Accordingly, in order to solve the data imbalance issue in aesthetic assessment, in this paper, we focus on training the convolutional neural network (CNN)-based aesthetic assessment model by dropping out the unavailable samples scored in the range of [3,5] from the training data set repetitively to overcome the effect of imbalanced aesthetic data on classification. Further, the aesthetic highlight region of photo images are extracted by subtracting two specific feature maps of first convolutional layers of two repetitively trained models, to analyze the correlations of the highlight regions with the aesthetic assessments, and explain what aesthetic information influence the aesthetic quality of the photos. Furthermore, the source of testing data set is different from one of the training data set. It was collected from the recommended photos of 500px which is an online photography network. The experimental results show that the proposed method is effective.

Related work
The photos' aesthetic level assessment exhibit highly-skewed score distribution as shown in Fig. 1. As described in [9], for such class-imbalanced data, the minority class often contains very few instances with high degree of visual variability. The scarcity and high variability make the genuine neighborhood of these instances easy to be invaded by other imposter nearest neighbors. In [11], a comprehensive literature survey to tackle the class data imbalance problem was reviewed. Generally, there are two groups of solutions: data re-sampling and cost-sensitive learning. The former group focuses on learning equally good classifiers by random under-sampling and over-sampling techniques. The latter group operates at the algorithmic level by adjusting misclassification. A well-known issue with over-sampling is its tendency to overfitting.
Therefore, under-sampling is often preferred, although potentially valuable information may be removed. Cost-sensitive alternatives avoid these problems by directly imposing heavier penalty on misclassifying the minority class. In [9], a data structure-aware deep learning approach with build-in margins for imbalanced classification was proposed.
However, these literature methods mainly aim at the classification of the classes which are defined explicitly. For the classification of aesthetic assessment scores which are of ambiguousness, the state-of-art methods seem not to be available.

Training photo aesthetic assessment model
In this paper, we propose a CNN-based learning method for photo Aesthetic assessment model by repetitively dropping out the low likelihood samples of majority score classes, so as to ameliorate the invasion to the minority score classes, and the loss of valuable features of majority instances. The idea behind is the assumption that the instance with low likelihood to the majority score classes is what is ambiguously assessed.
These samples are easy to overwhelm the Aesthetic assessment model. The system diagram is shown in Fig. 2. . The value of of the sample could be treated as its likelihood belong to . Based on , the instances, which apply to be unavailable for the network learning, are dropped out from the original training data set. The idea behind is that the neighboring scores are mutually related, and an instance labeled with the score of the majority classes while having the low likelihood to them may become the imposter nearest neighbors of the minority classes to invade the genuine neighborhood of those.
Accordingly, the conditions of dropping-out the unavailable samples are expressed by the expressions (1), (2), and (3).
For a sample 2 , For a sample 3 , Where, the score class having most samples is denoted 1 , the next is 2 , the third is as 3 . The sample which is labeled with Then, based on the previously trained aesthetic assessment network ℎ 1 , the network is retrained by transfer learning with the new training data set dropping out the unavailable instances. The accordingly generated network is called ℎ 2 . Based on ℎ 2 , the likelihood of all samples of the original training data set belonging to each score class is calculated, and the classification is executed. The samples are dropped out if they apply to the above removing conditions. It is noticed that the samples that are dropped out based on the previously trained network could be remained based on the current network so as to void the sample's missremoving to loss the valid information.
Repetitively, based on the latest trained aesthetic assessment network ℎ −1 , the network is retrained by transfer learning with the latest training data set dropping out the unavailable instances. Using the latest generated network ℎ , the likelihood of all samples of the original training data set belonging to each score class is calculated, and classification is executed. The samples are dropped out if they apply to the dropping out conditions. This learning procedure is continued until the value of loss function is less than a threshold Loss. Here, the loss function is defined as a quadratic loss function regarding labeling and assigned scores of instances.
The finally generated photo aesthetic assessment network is called ℎ .

Extracting aesthetics highlight region
In this section, we proposed a method of extracting aesthetic highlight regions of photos by using the repetitively trained aesthetic assessment networks, so as to analyze how the photos are assessed by the pro-photographer to investigate the composition features of good photos, and correlations the salient objects with backgrounds in the photos. The diagram of the method is shown in Fig. 3. . The index J of the corresponding feature maps which are of the minimal correlation is identified by the following equation.
Where, corr indicates the correlation. The difference map of the two feature maps with index J could be calculated by subtracting conv1 and conv1 − , the equation of which is as the following. Moreover, the different maps of above two instances regarding conv1 2 and conv1 1 is shown in Fig.5. conv1 1 is the feature map of ℎ 1 which is firstly trained based on alexNet by the original data set. The left is the result of the upper instance, and the right corresponds to the lower instance. Obviously, the highlight regions aren't emerged explicitly in these two difference maps. So, it is indirectly validated the effectiveness of repetitively training network by dropping out the unavailable instances.

Data set
Although the AVA data set [12] is the largest publicly available aesthetics dataset providing over 250,000 images in total, each image in which was aesthetically assessed by about 200 people with the rating score ranging from 1 to 10, all of the images were finally labeled with the mean score that lost the individual`s aesthetic sense although the aesthetic tendencies of a human group could be reflected. However, embodying the aesthetic taste of a human being is important in training our aesthetic assessment model. Therefore, we conduct our training data set which contains 3100 photos assessed aesthetically by a professional photographer, which is called xiheAA. These photos were taken by the students of the photographer's class. The scores ranging from 2 to 9. The distribution of the scores is shown in Fig. 1. The class having most samples is score 4; the next is score 3; the third is score 5. Also, we collect testing data from the other source, 500px. 310 images were downloaded randomly from the 500px's recommended photo set.

Aesthetic level classification
In this section, we elaborate how the repetitively trained network models improve the aesthetic level classification performance. The method proposed in section 3 is implemented on Matlab. The function trainNetwork is used to fine-tune the hyperparameters of the network based on transfer learning. The function classify is used to classify the samples based on the trained network. Table 1 illustrates the different score class's classification accuracies of the samples in the original xiheAA data set against the different networks. As described above, ℎ 1 is the network fine-tuning the hyper-parameters of alexNet by transfer learning using the original xiheAA data set. ℎ (i>=2) is the network finetuning the hyper-parameters of ℎ −1 by transfer learning using the new xiheAA data set from which the samples in the classes of score 3, score 4 and score 5 that  Table 1 Score's Classification accuracies against different networks It is observed from the results shown in Table 1 that the classification accuracies of minority classes with salient features, such as the class of score 2 indicating poor aesthetic quality, and the classes of score 7, score 8, score 9 indicating excellent quality, are improved dramatically by using the repetitively trained networks. In the cases of score 2, score 8 and score 9, the accuracies increase gradually from low levels to 100%.
In the case of score 7 with the relatively high number of samples, the accuracies are improved from 33% to 86%. For the relatively majority classes of score3 and score 5, the classification accuracies also ameliorate gradually from low levels to high levels. For the class of score 4 with the dominant samples, the classification accuracy is 94% by using the initial network ℎ 1 . Then the accuracy decreases to 78% using the first retrained network ℎ 2 . And then, the accuracies increases gradually using the further retrained networks. The accuracy becomes 91% using the retrained network ℎ 10 , although this doesn't reach to the initial accuracy 94%. The changing tendency of the accuracies is reasonable because removing the unavailable instances in the domination classes in training ameliorates the invasion of these samples to the minority classes so as to achieve the balance of the classification accuracies against the different classes. Table 2 shows the Loss values against the repetitively trained different networks.

Table 2 Loss values against different networks
It is observed that the loss value decrease gradually with repetitively training the network. The change tendency corresponds to the increasing of the classification accuracy shown in Table 1.
Next, the testing data set that were downloaded randomly from the 500px's recommended photo set is used to verity the effectiveness of the proposed method. Because these samples were selected by pro-photographers, it is thought that the qualities of aesthetic level are relatively high. Many of them should be over the fair levels, although they are not scored. So, the assigned rate is conducted to validate the model's aesthetic level classification performance against such testing data set. The assigned rate is defined by equation (6).
Where, # indicates the number, # indicates the number of photo images assigned to class i, and # indicates the number of photos in the data set. Table 3 shows the assigned rates of the samples in the testing data set using the different networks. ℎ 1 is the initial aesthetic assessment network, and ℎ (i>=2) is the retrained network by the data set dropping out the unavailable instances. Table 3 illustrates the assigned rates of the samples to the different score classes. Table 3 assigned rates of the samples It is observed that the assigned rates of the samples to the score classes is dramatically changed by using the retrained networks, especially to the classes with salient features.
For example, for the class of score 7 indicating the excellent aesthetic level, the assigned rate by using the network ℎ 10 is about four times to one by using the initial ℎ 1 . For the classes of fair level with the score less than 5, the assigned rates decrease about 50%. For the classes of score 5 and score 6 indicating good level, the rates increase about 200%. The total assigned rate of the classes with the score over 4 increases 238% from 0.24 to 0.57. Such result is acceptable, because it is accordant to the above assumption that the aesthetic quality of most of the samples in the testing data set should be good.
On the other hand, some samples assigned to the classes of score 3 and score 7 are shown in Fig. 6. The upper images are assigned to the score 3, and the lower images are assigned to score 7. It is obviously that the visual aesthetic quality of the lower images is better than one of the upper images, and seems to meet the common techniques for composing a good photo. So, the classification results of these samples are reliable. Moreover, it make us know that the fair photos exist in the above testing data set although they are selected by the pro-photographers. Accordingly, it is verified that the results shown in Table 3 are acceptable.
As a whole, the experimental results demonstrates that the proposed method for the photo's aesthetic level classification is effective.

Aesthetic analysis of highlight region
In section 4, extracting aesthetics highlight region from the photo image by using the repetitively trained networks was proposed. In this section, we focus on analyzing the correlation of the extracted region with the aesthetic assessment, so as to illustrate how to improve the photo's aesthetic quality.    It is observed that the extracted parts look plain and dull. There are not the salient objects in the extracted region. So, we know that the photos without salient objects often obtain the fair assessment. Fig. 9 shows the highlight regions of some photos labeled with score 7 in the training data set. Fig. 9 shows the highlight regions of some photos labeled with score 7 It is observed that the extracted parts are very clear. The salient objects are distinctive and made outstanding, and look pretty. So, we understand that the distinctive object with the clear highlight region make the photo have the high aesthetic quality.
Accordingly, we can say that the photos' aesthetics highlight regions extracted by using the repetitively trained aesthetic assessment network reveal the photo's aesthetic qualities. By analyzing the compositions of the extracted elements with the aesthetic scores assigned to the photos, it is possible to learn how to arrange the elements in the photo to make up a good photo.

Conclusion
In this paper, we propose a method of learning the aesthetic assessment model with the imbalance data set. Using CNN and transfer learning, the aesthetic network was repetitively trained by dropping out the unavailable instances in the training data set.
The mechanism of relearning and the conditions of removing was presented, and how the repetitively trained network models improve the aesthetic level classification performance was analyzed. Moreover, based on the feature maps of the retrained models, a method of extracting aesthetics highlight region of the photo was described, and how such region correlates to the aesthetic assessment was illustrated. Overall, the system could not only estimates the photo's aesthetic level, but also indicate why this estimation was induced. We think that the proposed method is also available for other domains which are relevant to the subjective assessment.