Question: represent the text to find a very short summary (1-2 sentences max).

Question: Self-supervised learning (SlfSL), aiming at learning feature representations through ingeniously designed pretext tasks without human annotation, has achieved compelling progress in the past few years. Very recently, SlfSL has also been identified as a promising solution for semi-supervised learning (SemSL) since it offers a new paradigm to utilize unlabeled data. This work further explores this direction by proposing a new framework to seamlessly couple SlfSL with SemSL. Our insight is that the prediction target in SemSL can be modeled as the latent factor in the predictor for the SlfSL target. Marginalizing over the latent factor naturally derives a new formulation which marries the prediction targets of these two learning processes. By implementing this framework through a simple-but-effective SlfSL approach -- rotation angle prediction, we create a new SemSL approach called Conditional Rotation Angle Prediction (CRAP). Specifically, CRAP is featured by adopting a module which predicts the image rotation angle \textbf{conditioned on the candidate image class}. Through experimental evaluation, we show that CRAP achieves superior performance over the other existing ways of combining SlfSL and SemSL. Moreover, the proposed SemSL framework is highly extendable. By augmenting CRAP with a simple SemSL technique and a modification of the rotation angle prediction task, our method has already achieved the state-of-the-art SemSL performance. The recent success of deep learning is largely attributed to the availability of a large amount of labeled data. However, acquiring high-quality labels can be very expensive and time-consuming. Thus methods that can leverage easily accessible unlabeled data become extremely attractive. Semisupervised learning (SemSL) and self-supervised learning (SlfSL) are two learning paradigms that can effectively utilize massive unlabeled data to bring improvement to predictive models. SemSL assumes that a small portion of training data is provided with annotations and the research question is how to use the unlabeled training data to generate additional supervision signals for building a better predictive model. In the past few years, various SemSL approaches have been developed in the context of deep learning. The current state-of-the-art methods, e.g. MixMatch , unsupervised data augmentation , converge to the strategy of combining multiple SemSL techniques, e.g. Π-Model , Mean Teacher , mixup , which have been proved successful in the past literature. SlfSL aims for a more ambitious goal of learning representation without any human annotation. The key assumption in SlfSL is that a properly designed pretext predictive task which can be effortlessly derived from data itself can provide sufficient supervision to train a good feature representation. In the standard setting, the feature learning process is unaware of the downstream tasks, and it is expected that the learned feature can benefit various recognition tasks. SlfSL also offers a new possibility for SemSL since it suggests a new paradigm of using unlabeled data, i.e., use them for feature training. Recent work has shown great potential in this direction. This work further advances this direction by proposing a new framework to seamlessly couple SlfSL with SemSL. The key idea is that the prediction target in SemSL can serve as a latent factor in the course of predicting the pretext target in a SlfSL approach. The connection between the predictive targets of those two learning processes can be established through marginalization over the latent factor, which also implies a new framework of SemSL. The key component in this framework is a module that predicts the pretext target conditioned on the target of SemSL. In this preliminary work, we implement this module by extending the rotation angle prediction method, a recently proposed SlfSL approach for image recognition. Specifically, we make its prediction conditioned on each candidate image class, and we call our method Conditional Rotation Angle Prediction (CRAP). The proposed framework is also highly extendable. It is compatible with many SemSL and SlfSL approaches. To demonstrate this, we further extend CRAP by using a simple SemSL technique and a modification to the rotation prediction task. Through experimental evaluation, we show that the proposed CRAP achieves significantly better performance than the other SlfSL-based SemSL approaches, and the extended CRAP is on par with the state-of-the-art SemSL methods. In summary, the main contributions of this paper are as follows: • We propose a new SemSL framework which seamlessly couples SlfSL and SemSL. It points out a principal way of upgrading a SlfSL method to a SemSL approach. • Implementing this idea with a SlfSL approach, we create a new SemSL approach (CRAP) that can achieve superior performance than other SlfSL-based SemSL methods. • We further extend CRAP with a SemSL technique and an improvement over the SlfSL task. The ed new method achieves the state-of-the-art performance of SemSL. Our work CRAP is closely related to both SemSL and SlfSL. SemSL is a long-standing research topic which aims to learn a predictor from a few labeled examples along with abundant of unlabeled ones. SemSL based on different principals are developed in the past decades, e.g., "transductive" models , multi-view style approaches and generative model-based methods , etc. Recently, the consistency regularization based methods have become quite influential due to their promising performance in the context of deep learning. Specifically, Π-Model requires model's predictions to be invariant when various perturbations are added to the input data. Mean Teacher enforces a student model producing similar output as a teacher model whose weights are calculated through the moving average over the weight of student model. Virtual Adversarial Training encourages the predictions for input data and its adversarially perturbed version to be consistent. More recently, mixup has emerged as a powerful SemSL regularization method which requires the output of mixed data to be close to the output mixing of original images. In order to achieve good performance, most state-of-the-art approaches adopt the strategy of combining several existing techniques together. For example, Interpolation Consistency Training incorporates Mean Teacher into the mixup regularization, MixMatch adopts a technique that uses fused predictions as pseudo prediction target as well as the mixup regularization. Unsupervised data augmentation upgrades Π-Model with advanced data augmentation methods. SlfSL is another powerful paradigm which learns feature representations through training on pretext tasks whose labels are not human annotated. Various pretext tasks are designed in different approaches. For example, image inpainting trains model to reproduce an arbitrary masked region of the input image. Image colorization encourages model to perform colorization of an input grayscale image. Rotation angle prediction forces model to recognize the angle of a rotated input image. After training with the pretext task defined in a SlfSL method, the network is used as a pretrained model and can be fine-tuned for a downstream task on task-specific data. Generally speaking, it is still challenging for SlfSL method to achieve competitive performance to fully-supervised approaches. However, SlfSL provides many new insights into the use of unlabeled data and may have a profound impact to other learning paradigms, such as semi-supervised learning. SlfSL based SemSL is an emerging approach which incorporates SlfSL into SemSL. The most straightforward approach is to first perform SlfSL on all available data and then fine-tune the learned model on labeled samples. S 4 L ) is a newly proposed method which jointly train the downstream task and pretext task in a multi-task fashion without breaking them into stages. In this paper, we further advance this direction through proposing a novel architecture which explicitly links these two tasks together and ensure that solving one task is beneficial to the other. In SemSL, we are given a set of training samples {x 1, x 2, · · ·, x n} ∈ X with only a few of them X l = {x 1, x 2, · · ·, x l} ∈ X annotated with labels {y 1, y 2, · · ·, y l} ∈ Y l (usually l << n and y is considered as discrete class label here). The goal of a SemSL algorithm is to learn a better posterior probability estimator over y, i.e., p(y|x, θ) with θ denoting model parameters, from both labeled and unlabeled training samples. SlfSL aims to learn feature representations via a pretext task. The task usually defines a target z, which can be derived from the training data itself, e.g., rotation angle of the input image. Once z is defined, SlfSL is equivalent to training a predictor to model p(z|x; θ). There are two existing schemes to leverage SlfSL for SemSL. The first is to use SlfSL to learn the feature from the whole training set and then fine-tuning the network on the labeled part. The other is jointly optimizing the tasks of predicting y and z, as in the recently proposed S 4 L method. As shown in Figure 1 (a), S 4 L constructs a network with two branches and a shared feature extractor. One branch for modeling p(y|x; θ) and another branch for modeling p(z|x; θ). However, in both methods the pretext target z predictor p(z|x; θ) is implicitly related to the task of predicting y. Our framework is different in that we explicitly incorporate y into the predictor for z. Specifically, we treat y as the latent factor in p(z|x; θ) and factorize p(z|x; θ) through marginalization: Eq. 1 suggests that the pretext target predictor p(z|x; θ) can be implemented as two parts: a model to estimate p(y|x; θ) and a model to estimate z conditioned on both x and y, i.e., p(z|x, y; θ). For the labeled samples, the ground-truth y is observed and can be used for training p(y|x; θ). For unlabeled samples, the estimation from p(y|x; θ) and p(z|x, y; θ) will be combined together to make the final prediction about z. Consequently, optimizing the loss for p(z|x; θ) will also provide gradient to back-propagate through p(y|x; θ). This is in contrast to the case of S 4 L, where the gradient generated from the unlabeled data will not flow through p(y|x; θ). Theoretically, p(z|x; θ) and p(y|x; θ) can be two networks, but in practise we model them as two branches connecting to a shared feature extractor. p(z|x; θ) suggested by Eq. 1 is essentially a pretext target predictor with a special structure and partial observations on its latent variable, i.e. y. The benefits of using such a predictor can be understood from three perspectives: p(y|x; θ) in Eq. 1 acts as a soft selector to select p(z|x, y; θ) for predicting z. If the estimation of p(y|x; θ) is accurate, it will select p(z|x, y =ŷ(x); θ) for prediction and update, whereŷ(x) is the true class of x. This selective updating will make p(z|x, y; θ) give more accurate prediction over z if y matchesŷ(x). After such an update, p(z|x, y; θ) will in turn encourage p(y|x; θ) to attain higher value for y =ŷ(x) since the prediction from p(z|x, y =ŷ(x); θ) is more likely to be accurate. Thus, the terms p(y|x; θ) and p(z|x, y; θ) will reinforce each other during training. even if p(y|x; θ) is not accurate (this may happen at the beginning of the training process), p(z|x, y; θ) can still perform the pretext target prediction and act as an unsupervised feature learner. Thus, the features will be gradually improved in the course of training. With a better feature representation, the estimation of p(y|x; θ) will also be improved. Finally, to predict z in Eq. 1, p(z|x, y; θ) needs to be evaluated for each candidate y. This in effect is similar to creating an ensemble of diversified pretext target predictors and with the combination weight given by p(y|x; θ) according to the marginalization rule. Thus, training features with Eq. 1 may enjoy the benefit from ensemble learning. Again, this will lead to better features and thus benefit the modelling of p(y|x; θ) and p(z|x, y; θ). The above framework provides a guideline for turning a SlfSL method into a SemSL algorithm: modifying a SlfSL predictor p(z|x; θ) by p(z|x, y; θ) and introducing a branch for p(y|x; θ) optimizing the prediction of z on the SemSL dataset and update the branches p(z|x, y; θ), p(y|x; θ) and their shared feature extractor. using p(y|x; θ) as downstream task predictor or adding an additional branch for training p(y|x; θ) only with the labeled data as in S 4 L. More details about the additional branch will be explained in Section 4. < l a t e x i t s h a 1 _ b a s e 6 4 = " t 2 W 6 x u / Y c G q z 7 e 6 w r U f 4 A 7 6 z e 2 A = " < l a t e x i t s h a 1 _ b a s e 6 4 = " t 2 W 6 x u / Y c G q z 7 e 6 w r U f 4 A 7 6 z e 2 A = " In the following part, we will describe an implementation of this framework, which is realized by upgrading the rotation-angle prediction-based SlfSL to its conditional version. Rotation angle prediction is a recently proposed SlfSL approach for image recognition. It randomly rotates the input image by one of the four possible rotation angles ({0 •, 90 •}) and requires the network to give a correct prediction of the rotation angle. Despite being extremely simple, this method works surprisingly well in practice. The underlying logic is that to correctly predict the rotation angle, the network needs to recognize the canonical view of objects from each class and thus enforces the network to learn informative patterns of each image category. Following the proposed framework, we upgrade rotation angle prediction to conditional rotation angle prediction (CRAP) for semi-supervised learning. In this case, z in Eq. 1 is the rotation angle and y is the class label of input image x. We realize p(z|x, y; θ) by allocating a rotation angle prediction branch for each class. The prediction from each branch is then aggregated with the aid of p(y|x; θ) for the final prediction of z as shown in Eq. 1. A more detailed schematic illustration of the CRAP method is shown in Figure 1 (b). As seen, our method adopts a network with multiple branches and a shared feature extractor. Specifically, branches within the dashed box are called auxiliary branches since they are only used for training and will be discarded at the test stage. It contains C rotation predictors which corresponds to p(z|x, y; θ) and a semantic classifier which generates p(y|x; θ). The auxiliary branches and feature extractor are trained by using the procedure described in Section 3. Note that in CRAP, we do not directly use the semantic classifier from the auxiliary branches as the final classifier. Instead, we introduce an additional semantic classifier and learn it only via the loss incurred from the labeled data. This treatment is similar to S 4 L and we find this strategy work slightly better in practice. We postulate the reason is that the p(y|x; θ) branch in auxiliary branches is mainly trained by the supervision generated from the optimization of p(z|x; θ). Such supervision is noisy comparing with the loss generated from the ground-truth y. It is better to use such a branch just for feature training since the latter is more tolerant to noisy supervision. Remark: One potential obstacle of our model is that the quantity of parameters in the auxiliary branches would increase significantly with a large C. To tackle this, we propose to perform dimension reduction for the features feeding into the rotation predictor. Results in Section 5.3 show that this scheme is effective as our performance will not drop even when the dimension is reduced from 2048 to 16. The CRAP method is also highly expendable. In the following, we will extend CRAP from two perspectives: improving p(y|x; θ) and improving p(z|x, y; θ). As discussed in Section 3, our method essentially introduces a network module with a special structure and partial observations on the latent variable y. Besides using labeled data to provide supervision for y, we can also use existing SemSL techniques to provide extra loss for modeling p(y|x; θ). To implement such an extension, we employ a simple SemSL loss as follows: we rotate each image in four angles within one batch (the prediction of the rotated image can be obtained as the byproduct of CRAP) and obtain the arithmetic averagep of the predicted distributions across these four rotated samples. Then we perform a sharpening operation overp as in MixMatch: where C is the number of classes and T ∈ is a temperature hyper-parameter. Then we use the cross entropy betweenp i and p(y|x; θ) (in auxiliary branches) as an additional loss. Note that other (more powerful) SemSL can also apply here. We choose the above SemSL technique is simply because its operation, i.e. image rotation, has already been employed in the CRAP algorithm and thus could be reused to generate the additional SemSL loss without increasing the complexity of the algorithm. We also make another extension over CRAP by introducing an improved version of the conditional rotation prediction task. Specifically, we require the rotation prediction branch to predict rotation angle for a mixed version of the rotated image, that is, we randomly mix the input image x i with another randomly sampled rotated image x j via x mix = αx i + (1 − α)x j, with α sampled from [0.5, 1]. Meanwhile, the class prediction p(y|x i ; θ) is calculated from the unmixed version of the input x i. In such a design, the network needs to recognize the rotation angle of the target object with the noisy distraction from another image, and we call this scheme denoising rotation prediction. The purpose of introducing this modified task is to make the SlfSL task more challenging and more dependent on the correct prediction from p(y|x; θ). To see this point, let's consider the following example. Letter'A' is rotated with 270 • and is mixed with letter'B' with rotation 90 •. Directly predicting the rotation angle for this mixed image encounters an ambiguity: whose rotation angle, A's or B's, is the right answer? In other words, the network cannot know which image class is the class-of-interest. This ambiguity can only be resolved from the output of p(y|x; θ) since its input is unmixed target image. Therefore, this improved rotation prediction task relies more on the correct prediction from the semantic classifier and training through CRAP is expected to give stronger supervision signal to p(y|x; θ). Note that although the denoising rotation prediction also uses mix operation, it is completely different from mixup. The latter constructs a loss to require the output of the mixed image to be mixed version of the outputs of original images. This loss is not applied in our method. For more algorithm details about CRAP and the extended CRAP, please refer to the Appendix A.1. In this section, we conduct experiments to evaluate the proposed CRAP method 1. The purpose of our experiments is threefolds: to validate if CRAP is better than other SlfSL-based SemSL algorithms. to compare CRAP and extended CRAP (denoted as CRAP+ hereafter) against the state-of-the-art SemSL methods. to understand the contribution of various components in CRAP. To make a fair comparison to recent works, different experimental protocols are adopted for different datasets. Specifically, for CIFAR-10 and CIFAR-100 and SVHN , we directly follow the settings in . For ILSVRC-2012 (, our settings are identical to except for data pre-processing operations for which we only use the inception crop augmentation and horizontal mirroring. We ensure that all the baselines are compared under the same setting. Followed the standard settings of SemSL, the performance with different amount of labeled samples are tested. For CIFAR-10 and SVHN, sample size of labeled images is ranged in five levels: {250, 500, 1000, 2000, 4000}. For CIFAR-100, 10000 labeled data is used for training. For ILSVRC-2012, 10% and 1% of images are labeled among the whole dataset. In each experiment, three independent trials are conducted for all datasets except for ILSVRC-2012. See more details in Table 8 in Appendix. Firstly, we compare CRAP to other SlfSL-based SemSL algorithms on five datasets: CIFAR-10, CIFAR-100, SVHN, SVHN+Extra and ILSVRC-2012. Two SlfSL-based SemSL baseline approaches are considered: 1) Fine-tune: taking the model pretrained on the pretext task as an initialization and fine-tuning with a set of labeled data. We term this method Fine-tune in the following sections. 2) S 4 L: S 4 L method proposed in. Note that we do not include any methods which combine other SemSL techniques. For this reason, we only use our basic CRAP algorithm in the comparison in this subsection. As a reference, we also report the performance obtained by only using the labeled part of the dataset for training, denoting as Labeled-only. The experimental are as follows: The are presented in Table 1. We find that the "Fine-tune" strategy leads to a mixed amount of improvement over the "Labeled-only" case. It is observed that a large improvement can be obtained when the amount of labeled samples is ranged from 500 to 2000 but not on 250 and 4000's settings. It might be because on one hand too few labeled samples are not sufficient to perform an effective fine-tuning while on the other hand the significant improvement diminishes after the sample size increase. In comparison, S 4 L achieves much better accuracy for the case of using few samples. This is largely benefited from its down-stream-task awareness design -the labeled training samples exerts impact at the feature learning stage. Our CRAP method achieves significantly better performance than those two ways of incorporating SlfSL for SemSL and always halves the test error of S 4 L in most cases. Table 2 shows the of each method. Somehow surprisingly, we find that the Fine-tune and S 4 L do not necessarily outperform the Labeled-only baseline. They actually performs worse than Labeled-only on SVHN. With more training data in SVHN + Extra, S 4 L tends to bring benefits for enhancing performance when the size of labeled samples are small e.g., with 250 samples. In comparison, the proposed CRAP still manages to produce significant improvement over Labeled-only in all those settings. This clearly demonstrates that the simple combination of SlfSL and SemSL may not necessarily bring improvement and a properly-designed strategy of incorporating SlfSL with SemSL is crucial. CIFAR-100 As shown in Table 3, it is obvious that all SlfSL-based SemSL methods can have better accuracy than that of Labeled-only. S 4 L leads to a marginal improvement over Fine-tune although its performance is a little bit unstable on different partitions as shown by its higher variance. Again, the proposed CRAP achieves significant improvement over those baselines. Table 4 presents the of each method. The top block of Table 4 shows the reported in the original S 4 L paper and we also re-implement S 4 L based on the code of. Due to the difference of data pre-processing, in the upper block cannot be directly compared to those below. Again, we have observed that CRAP is consistently superior to S 4 L in all settings. As mentioned in Section 4, for saving the computational cost, we propose to reduce the dimensionality of features fed into the rotation angle predictor when there is a large number of classes. In Table 5, we demonstrates the effect of this scheme. As seen, the test performance stays the same when the feature dimensions is gradually reduced from 2048 to only 16 dimensions. This clearly validates the effectiveness of the proposed scheme. In the following section, we proceed to demonstrate the performance of CRAP+, that is, the extended CRAP method by incorporating the two extensions discussed in Section 4.1 and 4.2. We compare its performance against the current state-of-the-art methods in SemSL. Similar to , several SemSL baselines are considered: Pseudo-Label, Π-Model, Mean Teacher, Virtual Adversarial Training (VAT), MixUp and MixMatch 2. Since a fair and comprehensive comparison has been done in and we strictly follow the same experimental setting, we directly compare CRAP+ to the numbers reported in . The experimental are shown in Figure 2, Figure 3 and Table 6. As seen from those Figures and Table, the proposed CRAP+ is on-par with the best performed approaches, e.g., Mixmatch, in those datasets. This clearly demonstrates the power of the proposed method. Note that the current state-ofthe-art in SemSL is achieved by carefully combining multiple existing successful ideas in SemSL. In contrast, our CRAP+ achieves excellent performance via an innovative framework of marrying SlfSL with SemSL. Conceptually, the latter enjoys greater potential. In fact, CRAP might be further extended by using more successful techniques in SemSL, such as MixUp. Since the focus of this paper is to study how SlfSL can benefit SemSL, we do not pursue this direction here. Since there are several components in CRAP and CRAP+, we study the effect of adding or removing some components in order to provide additional insight into the role of each part. Specifically, we measure the effect of only adding extension 1 to CRAP, i.e., incorporating an additional SemSL loss through sharpening operations on the semantic classifier in auxiliary branches further adding extension 2 to CRAP. The ed model is identical to CRAP+ removing semantic classifier of main branch from CRAP. This is equivalent to using the semantic classifier in auxiliary branches for testing removing rotation angle prediction branch from auxiliary branches and adding extension 1 to CRAP. The ed structure can be seen as a variant of only using the SemSL technique in Extension 1 (but also with the classifier in main branch) removing whole auxiliary branches from CRAP, i.e., pure supervised method with data rotated. We conduct ablation studies on CIFAR-10 with 250 and 4000 labels with presented in Table 7. The main observations are: The two extensions in CRAP+ will bring varying degrees of improvement. Extension 1 in Section 4.1, i.e., a stronger p(y|x; θ) modeling, perhaps leads to greater improvement. Using an additional semantic classifier leads to a slight performance improvement over the strategy of directly utilizing p(y|x; θ) in the auxiliary branches for testing (method in third line from the bottom). Using the sharpening strategy as in our extension 1 and training a SemSL method alone does not produce good performance. This indicates the superior performance of CRAP+ is not simply coming from a strong SemSL method but its incorporation with the CRAP framework. Applying rotation as a data augmentation for labeled data (the last method in Table 7) will not lead to improved performance over the labeled-only baseline as by cross referring the in Table 9. This shows that the advantage of CRAP is not coming from the rotation data augmentation. In this work, we introduce a framework for effectively coupling SemSL with SlfSL. The proposed CRAP method is an implementation of this framework and it shows compelling performance on several benchmark datasets compared to other SlfSL-based SemSL methods. Furthermore, two extensions are incorporated into CRAP to create an improved method which achieves comparable performance to the state-of-the-art SemSL methods.

represent the tldr for finding the corresponding scientific article
E.g. The notion of order learning is proposed and it is applied to regression problems in computer vision == We propose order learning to determine the order graph of classes, representing ranks or priorities, and classify an object instance into one of the classes. To this end, we design a pairwise comparator to categorize the relationship between two instances into one of three cases: one instance is `greater than,' `similar to,' or `smaller than' the other. Then, by comparing an input instance with reference instances and maximizing the consistency among the comparison , the class of the input can be estimated reliably. We apply order learning to develop a facial age estimator, which provides the state-of-the-art performance. Moreover, the performance is further improved when the order graph is divided into disjoint chains using gender and ethnic group information or even in an unsupervised manner. To measure the quality of something, we often compare it with other things of a similar kind. Before assigning 4 stars to a film, a critic would have thought, "It is better than 3-star films but worse than 5-stars." This ranking through pairwise comparisons is done in various decision processes . It is easier to tell the nearer one between two objects in a picture than to estimate the distance of each object directly (; a). Also, it is easy to tell a higher pitch between two notes, but absolute pitch is a rare ability . Ranking through comparisons has been investigated for machine learning. In learning to rank (LTR), the pairwise approach learns, between two documents, which one is more relevant to a query . Also, in ordinal regression , to predict the rank of an object, binary classifications are performed to tell whether the rank is higher than a series of thresholds or not. In this paper, we propose order learning to learn ordering relationship between objects. Thus, order learning is related to LTR and ordinal regression. However, whereas LTR and ordinal regression assume that ranks form a total order , order learning can be used for a partial order as well. Order learning is also related to metric learning . While metric learning is about whether an object is'similar to or dissimilar from' another object, order learning is about'greater than or smaller than.' Section 2 reviews this related work. In order learning, a set of classes, Θ = {θ 1, θ 2, · · ·, θ n}, is ordered, where each class θ i represents one or more object instances. Between two classes θ i and θ j, there are three possibilities: θ i > θ j or θ i < θ j or neither (i.e. incomparable). These relationships are represented by the order graph. The goal of order learning is to determine the order graph and then classify an instance into one of the classes in Θ. To achieve this, we develop a pairwise comparator that determines ordering relationship between two instances x and y into one of three categories: x is'greater than,''similar to,' or'smaller than' y. Then, we use the comparator to measure an input instance against multiple reference instances in known classes. Finally, we estimate the class of the input to maximize the consistency among the comparison . It is noted that the parameter optimization of the pairwise comparator, the selection of the references, and the discovery of the order graph are jointly performed to minimize a common loss function. Section 3 proposes this order learning. We apply order learning to facial age estimation. Order learning matches age estimation well, since it is easier to tell a younger one between two people than to estimate each person's age directly (; a). Even when we assume that age classes are linearly ordered, the proposed age estimator performs well. The performance is further improved, when classes are divided into disjoint chains in a supervised manner using gender and ethnic group information or even in an unsupervised manner. Section 4 describes this age estimator and discusses its . Finally, Section 5 concludes this work. Pairwise comparison: It is a fundamental problem to estimate the priorities (or ranks) of objects through pairwise comparison. In the classic paper, noted that, even when direct estimates of certain quantities are unavailable, rough ratios between them are easily obtained in many cases. Thus, he proposed the scaling method to reconstruct absolute priorities using only relative priorities. The scaling method was applied to monocular depth estimation (a) and aesthetic assessment (b). Ranking from a pairwise comparison matrix has been studied to handle cases, in which the matrix is huge or some elements are noisy (; ; ;). On the other hand, the pairwise approach to LTR learns, between two documents, which one is more relevant to a query (; ; ;). The proposed order learning is related to LTR, since it also predicts the order between objects. But, while LTR sorts multiple objects with unknown ranks and focuses on the sorting quality, order learning compares a single object x with optimally selected references with known ranks to estimate the rank of x. Ordinal regression: Ordinal regression predicts an ordinal variable (or rank) of an instance. Suppose that a 20-year-old is misclassified as a 50-year old and a 25-year old, respectively. The former error should be more penalized than the latter. Ordinal regression exploits this characteristic in the design of a classifier or a regressor. and , a conversion scheme was proposed to transform an ordinal regression problem into multiple binary classification problems. Ordinal regression based on this conversion scheme has been used in various applications, including age estimation (; ;) and monocular depth estimation . Note that order learning is different from ordinal regression. Order learning performs pairwise comparison between objects, instead of directly estimating the rank of each object. In age estimation, ordinal regression based on the conversion scheme is concerned with the problem, "Is a person's age bigger than a threshold θ?" for each θ. In contrast, order learning concerns "Between two people, who is older?" Conceptually, order learning is easier. Technically, if there are N ranks, the conversion scheme requires N − 1 binary classifiers, but order learning needs only a single ternary classifier. Moreover, whereas ordinal regression assumes that ranks form a total order, order learning can be used even in the case of a partial order . Metric learning: A distance metric can be learned from examples of similar pairs of points and those of dissimilar pairs . The similarity depends on an application and is implicitly defined by user-provided examples. If a learned metric generalizes well to unseen data, it can be used to enforce the desired similarity criterion in clustering , classification , or information retrieval . Both metric learning and order learning learn important binary relations in mathematics: metric and order . However, a metric decides whether an object x is similar to or dissimilar from another object y, whereas an order tells whether x is greater than or smaller than y. Thus, a learned metric is useful for grouping similar data, whereas a learned order is suitable for processing ordered data. Age estimation: Human ages can be estimated from facial appearance (Kwon & da). proposed the aging pattern subspace, and introduced biologically inspired features to age estimation. Recently, deep learning has been adopted for age estimation. proposed OR-CNN for age estimation, which is an ordinal regressor using the conversion scheme. proposed Ranking-CNN, which is another ordinal regressor. While OR-CNN uses a common feature for multiple binary classifiers, Ranking-CNN employs a separate CNN to extract a feature for each binary classifier. grouped adjacent ages via the group-n encoding, determined whether a face belongs to each group, and combined the to predict the age. proposed the mean-variance loss to train a CNN classifier for age estimation. proposed the deep regression forests for age estimation. developed a compact age estimator using the two-points representation. Also, proposed a continuity-aware probabilistic network for age estimation. Figure 1: Examples of order graphs, in which node n precedes node m (n → m), if n divides m. For clarity, self-loops for reflexivity and edges deducible from transitivity are omitted from the graphs. 3 ORDER LEARNING Let us first review mathematical definitions and concepts related to order. An order , often denoted by ≤, is a binary relation on a set Θ = {θ 1, θ 2, · · ·, θ n} that satisfies the three properties of In real-world problems, an order describes ranks or priorities of objects. For example, in age estimation, θ i ≤ θ j means that people in age class θ i look younger than those in θ j. We may use the symbol →, instead of ≤, to denote an order on a finite set Θ. Then, the order can be represented by a directed graph using elements in Θ as nodes. If θ i → θ j, there is a directed edge from node θ i to node θ j. The order graph is acyclic because of antisymmetry and transitivity. For example, for n, m ∈ N, let n → m denote that m is a multiple of n. Note that it is an order on any subset of N. Figure 1(a) is the graph representing this order on {1, . . ., 9}. Elements θ i and θ j are comparable if θ i → θ j or θ j → θ i, or incomparable otherwise. In Figure 1(a), 6 and 8 are incomparable. In age estimation, it is difficult to compare apparent ages of people in different ethnic groups or of different genders. An order on a set Θ is total (or linear) if all elements in Θ are comparable to one another. In such a case, Θ is called a linearly ordered set. In some real-world problems, orders are not linear. In this work, a subset Θ c of Θ is referred to as a chain, if Θ c is linearly ordered and also maximal, i.e. there is no proper superset of Θ c that is linearly ordered. In Figure 1 (a), nodes 1, 2, 4, and 8 form a chain. In Figure 1 (b), the entire set is composed of three disjoint chains. Let Θ = {θ 1, θ 2, · · ·, θ n} be an ordered set of classes, where each class θ i represents one or more object instances. For example, in age estimation, age class 11 is the set of 11-year-olds. The objective of order learning is to determine the order graph, such as Figure 1 (a) or (b), and categorize an object instance into one of the classes. However, in many cases, order graphs are given explicitly or obvious from the contexts. For example, in quality assessment, there are typically five classes (poor → satisfactory → good → very good → excellent), forming a single chain. Also, in age estimation, suppose that an algorithm first classifies a person's gender into female or male and then estimates the age differently according to the gender. In this case, implicitly, there are separate age classes for each gender, and the age classes compose two disjoint chains similarly to Figure 1 (b). Thus, in this subsection, we assume that the order graph is already known. Also, given an object instance, we assume that the chain to which the instance belongs is known. Then, we attempt to categorize the instance into one of the classes in the chain. Section 3.4 will propose the order learning in the case of an unknown order graph, composed of disjoint chains. Instead of directly estimating the class of each instance, we learn pairwise ordering relationship between two instances. Let Θ c = {0, 1, . . ., N − 1} be a chain, where N is the number of classes. Let x and y be two instances belonging to classes in Θ c. Let θ(·) denote the class of an instance. Then, x and y are compared and their ordering relationship is defined according to their class difference as where τ is a threshold. To avoid confusion, we use', ≈, ≺' for the instance ordering, while'>, =, <' for the class order. In practice, the categorization in∼ is performed by a pairwise comparator in Figure 2, which consists of a Siamese network and a ternary classifier (b). To train the comparator, only comparable instance pairs are employed. We estimate the class θ(x) of a test instance x by comparing it with reference instances y m, 0 ≤ m ≤ M − 1, where M is the number of references. The references are selected from training data such that they are from the same chain as x. Given x and y m, the comparator provides one of three categories', ≈, ≺' as a . Let θ be an estimate of the true class θ(x). Then, the consistency between the comparator and the estimate is defined as is the indicator function. The function φ con (x, y m, θ) returns either 0 for an inconsistent case or 1 for a consistent case. For example, suppose that the pairwise comparator declares x ≺ y m but θ − θ(y m) > τ. Then, φ con (x, y m, θ) = 0 · 1 + 0 · 0 + 1 · 0 = 0. Due to a possible classification error of the comparator, this inconsistency may occur even when the estimate θ equals the true class θ(x). To maximize the consistency with all references, we estimate the class of x bŷ which is called the maximum consistency (MC) rule. Figure 3 illustrates this MC rule. It is noted that', ≈, ≺' is not an mathematical order. For example, if θ(x) + 3 4 τ = θ(y) = θ(z) − 3 4 τ, then x ≈ y and y ≈ z but x ≺ z. This is impossible in an order. More precisely, due to the quantization effect of the ternary classifier in∼,', ≈, ≺' is quasi-transitive , and'≈' is symmetric but intransitive. We use this quasi-transitive relation to categorize an instance into one of the classes, on which a mathematical order is well defined. In the simplest case of 1CH, all classes form a single chain Θ c = {0, 1, . . ., N − 1}. For example, in 1CH age estimation, people's ages are estimated regardless of their ethnic groups or genders. We implement the comparator in Figure 2 is written within the box. For θ = 7, there are six inconsistent boxes. For θ = 9, there are 24 such boxes. In this example, θ = 7 minimizes the inconsistency, or equivalently maximizes the consistency. Therefore,θ MC (x) = 7. where T is the set of all training instances and R ⊂ T is the set of reference instances. First, we initialize R = T and minimize co via the stochastic gradient descent. Then, we reduce the reference set R by sampling references from T. Specifically, for each class in Θ c, we choose M/N reference images to minimize the same loss co, where M is the number of all references and N is the number of classes. In other words, the reliability score of a reference candidate y is defined as and the M/N candidates with the highest reliability scores are selected. Next, after fixing the reference set R, the comparator is trained to minimize the loss co. Then, after fixing the comparator parameters, the reference set R is updated to minimize the same loss co, and so forth. In the test phase, an input instance is compared with the M references and its class is estimated using the MC rule in. In KCH, we assume that classes form K disjoint chains, as in Figure 1 (b). For example, in the supervised 6CH for age estimation, we predict a person's age according to the gender in {female, male} and the ethnic group in {African, Asian, European}. Thus, there are 6 chains in total. In this case, people in different chains are assumed to be incomparable for age estimation. It is supervised, since gender and ethnic group annotations are used to separate the chains. The supervised 2CH or 3CH also can be implemented by dividing chains by genders only or ethnic groups only. The comparator is trained similarly to 1CH. However, in computing the comparator loss in, a training instance x and a reference y are constrained to be from the same chain. Also, during the test, the type (or chain) of a test instance should be determined. Therefore, a K-way type classifier is trained, which shares the feature extractor with the comparator in Figure 2 and uses additional fully-connected (FC) layers. Thus, the overall loss is given by where co is the comparator loss and ty is the type classifier loss. The comparator and the type classifier are jointly trained to minimize this overall loss. During the test, given an input instance, we determine its chain using the type classifier, and compare it with the references from the same chain, and then estimate its class using the MC rule in. This subsection proposes an algorithm to separate classes into K disjoint chains when there are no supervision or annotation data available for the separation. First, we randomly partition the training set Input: T = training set of ordinal data, K = # of chains, N = # of classes in each chain, and M = # of references in each chain 1: Partition T randomly into T0,..., TK−1 and train a pairwise comparator 2: From T k, select M/N references y with the highest reliability scores α k (y) 4: end for 5: repeat 6: for each instance x do Membership Update (T k) 7: Assign it to T k *, where k * = arg max k β k (x) subject to the regularization constraint 8: end for 9: Fine-tune the comparator and train a type classifier using T0,..., TK−1 to minimize = co + ty 10: Assign it to T k where k is its type classification 12: end for 13: From T k, select M/N references y with the highest reliability scores α k (y) 15: end for 16: until convergence or predefined number of iterations Output: Pairwise comparator, type classifier, reference sets R0,..., RK−1 to, the comparator loss co can be written as where R k ⊂ T k is the set of references for the kth chain, α k (y) = x∈T k j q xy j log p xy j is the reliability of a reference y in the kth chain, and β k (x) = y∈R k j q xy j log p xy j is the affinity of an instance x to the references in the kth chain. Note that β k (x) = − y∈R k D(q xy p xy) where D is the Kullback-Leibler distance . Second, after fixing the chain membership T k for each chain k, we select references y to maximize the reliability scores α k (y). These references form R k. Third, after fixing R 0,..., R K−1, we update the chain membership T 0,..., T K−1, by assigning each training instance x to the kth chain that maximizes the affinity score β k (x). The second and third steps are iteratively repeated. Both steps decrease the same loss co in. The second and third steps are analogous to the centroid rule and the nearest neighbor rule in the Kmeans clustering , respectively. The second step determines representatives in each chain (or cluster), while the third step assigns each instance to an optimal chain according to the affinity. Furthermore, both steps decrease the same loss alternately. However, as described in Algorithm 1, we modify this iterative algorithm by including the membership refinement step in lines 10 ∼ 12. Specifically, we train a K-way type classifier using T 0,..., T K−1. Then, we accept the type classification to refine T 0,..., T K−1. This refinement is necessary because the type classifier should be used in the test phase to determine the chain of an unseen instance. Therefore, it is desirable to select the references also after refining the chain membership. Also, in line 7, if we assign an instance x to maximize β k (x) only, some classes may be assigned too few training instances, leading to data imbalance. To avoid this, we enforce the regularization constraint so that every class is assigned at least a predefined number of instances. This regularized membership update is described in Appendix A. We develop an age estimator based on the proposed order learning. Order learning is suitable for age estimation, since telling the older one between two people is easier than estimating each person's age directly (; a). It is less difficult to distinguish between a 5-year-old and a 10-year-old than between a 65-yearold and a 70-year-old. Therefore, in age estimation, we replace the categorization based on the arithmetic difference in∼ with that based on the geometric ratio as follows. which represent'older,''similar,' and'younger.' The consistency in is also modified accordingly. There are 5 reference images for each age class within range in this work (M = 330, N = 66). Thus, a test image should be compared with 330 references. However, we develop a twostep approach, which does at most 130 comparisons but performs as good as the method using 330 comparisons. The two-step estimation is employed in all experiments. It is described in Appendix B. We align all facial images using SeetaFaceEngine and resize them into 256 × 256 × 3. Then, we crop a resized image into 224 × 224 × 3. For the feature extractors in Figure 2, we use VGG16 without the FC layers . They yield 512-channel feature vectors. Then, the vectors are concatenated and input to the ternary classifier, which has three FC layers, yielding 512-, 512-, and 3-channel vectors sequentially. The 3-channel vector is normalized to the softmax probabilities of the three categories', ≈, ≺.' In∼, τ age is set to 0.1. In KCH with K ≥ 2, the type (or chain) of a test image should be determined. Thus, we design a type classifier, which shares the feature extractor with the comparator. Similarly to the ternary classifier, the type classifier uses three FC layers, yielding 512-, 512-, and K-channel vectors sequentially. The comparator and the type classifier are jointly trained. To initialize the feature extractors, we adopt the VGG16 parameters pre-trained on ImageNet . We randomly initialize all the other layers. We update the parameters using the Adam optimizer . We set the learning rate to 10 −4 for the first 70 epochs. Then, we select 5 references for each age class. Using the selected references, we fine-tune the network with a learning rate of 10 −5. We repeat the reference selection and the parameter fine-tuning up to 3 times. In the case of unsupervised chains, we enforce the regularization constraint (line 7 in Algorithm 1). By default, for each age, all chains are constrained to be assigned the same number of training images. If there are L training images of θ-year-olds, the age classes θ in the K chains are assigned L/K images, respectively, according to the affinity scores β k (x) by Algorithm 2 in Appendix A. MORPH II is the most popular age estimation benchmark, containing about 55,000 facial images in the age range. IMDB-WIKI is another dataset containing about 500,000 celebrity images obtained from IMDB and Wikipedia. It is sometimes used to pre-train age estimation networks. Optionally, we also select 150,000 clean data from IMDB-WIKI to pre-train the proposed pairwise comparator. Although several facial age datasets are available, most are biased to specific ethnic groups or genders. Data unbalance restricts the usability and degrades the generalization performance. Thus, we form a'balanced dataset' from MORPH II, AFAD , and UTK (b). Table 1 shows how the balanced dataset is organized. Before sampling images from MORPH II, AFAD, and UTK, we rectify inconsistent labels by following the strategy in. For each combination of gender in {female, male} and ethnic group in {African, Asian, European}, we sample about 6,000 images. Also, during the sampling, we attempt to make the age distribution as For performance assessment, we calculate the mean absolute error (MAE) and the cumulative score (CS) . MAE is the average absolute error between predicted and ground-truth ages. Given a tolerance level l, CS computes the percentage of test images whose absolute errors are less than or equal to l. In this work, l is fixed to 5, as done in Chang et al. Table 2 compares the proposed algorithm (1CH) with conventional algorithms on MORPH II. As evaluation protocols for MORPH II, we use four different settings, including the 5-fold subjectexclusive (SE) and the 5-fold random split (RS) . Appendix C.1 describes these four settings in detail and provides an extended version of Table 2. OHRank, OR-CNN, and Ranking-CNN are all based on ordinal regression. OHRank uses traditional features, yielding relatively poor performances, whereas OR-CNN and Ranking-CNN use CNN features. DEX, DRFs, MO-CNN, MV, and BridgeNet employ VGG16 as backbone networks. Among them, MV and BridgeNet achieve the state-of-the-art , by employing the mean-variance loss and the gating networks, respectively. The proposed algorithm outperforms these algorithms in setting C, which is the most challenging task. Furthermore, in terms of CS, the proposed algorithm yields the best performances in all four settings. These outstanding performances indicate that order learning is an effective approach to age estimation. In Table 3, we analyze the performances of the proposed algorithm on the balanced dataset according to the number of hypothesized chains. We also implement and train the state-of-the-art MV on the balanced dataset and provide its using supervised chains. Let us first analyze the performances of the proposed algorithm using'supervised' chains. The MAE and CS scores on the balanced dataset are worse than those on MORPH II, since the balanced dataset contains more diverse data and thus is more challenging. By processing facial images separately according to the genders (2CH), the proposed algorithm reduces MAE by 0.05 and improves CS by 0.2% in comparison with 1CH. Similar improvements are obtained by 3CH or 6CH, which consider the ethnic groups only or both gender and ethnic groups, respectively. In contrast, in the case of MV, multi-chain hypotheses sometimes degrade the performances; e.g., MV (6CH) yields a lower CS than MV (1CH). Regardless of the number of chains, the proposed algorithm trains a single comparator but uses a different set of references for each chain. The comparator is a ternary classifier. In contrast, MV (6CH) should train six different age estimators, each of which is a 66-way classifier, to handle different chains. Thus, their training is more challenging than that of the single ternary classifier. Note that, for the multi-chain hypotheses, the proposed algorithm first identifies the chain of a test image using the type classifiers, whose accuracies are about 98%. In Table 3, these Comparison are color-coded. Cyan, yellow, and magenta mean that the test subject is older than , similar to (≈), and younger than (≺) a reference. The age is estimated correctly as 22. type classifiers are used to obtain the of the proposed algorithm, whereas the ground-truth gender and ethnic group of each test image are used for MV. Figure 4 shows how to estimate an age in 6CH. In this test, the subject is a 22-year-old Asian male. He is compared with the references who are also Asian males. Using the comparison , the age is correctly estimated as 22 by the MC rule in. Table 4 lists the MAE for each test chain. Europeans yield poorer MAEs than Africans or Asians. However, this is not due to inherent differences between ethnic groups. It is rather caused by differences in image qualities. As listed in Table 1, more European faces are sampled from UTK. The UTK faces were crawled from the Internet and their qualities are relatively low. Also, from the cross-chain test using 6CH, some observations can be made: • Except for the As-F test chain, the lowest MAE is achieved by the references in the same chain. • Eu-M and Eu-F are mutually compatible. For Eu-M, the second best performance is obtained by the Eu-F references, and vice versa. On the other hand, some chains, such as Af-M and Eu-F, are less compatible for the purpose of the proposed age estimation. Table 3 also includes the performances of the proposed algorithm using'unsupervised' chains. The unsupervised algorithm outperforms the supervised one, which indicates that the gender or ethnic group is not the best information to divide data for age estimation. As in the supervised case, 2CH, 3CH, and 6CH yield similar performances, which means that two chains are enough for the balanced set. Compared with MV (1CH), the unsupervised algorithm (2CH) improves the performances significantly, by 0.33 in terms of MAE and 4.1% in terms of CS. Figure 5 shows how training images are divided into two chains in the unsupervised 2CH. During the membership update, for each age, each chain is regularized to include at least a certain percentage (κ) of the training images. In the default mode, the two chains are assigned the same number of images with κ = 50%. However, Appendix C.3 shows that the performance is not very sensitive to κ. At κ = 10%, MAE = 4.17 and CS = 73.7%. From Figure 5, we observe • The division of the chains is not clearly related to genders or ethnic groups. Regardless of genders or ethnic groups, about half of the images are assigned to chain 1 and the others to chain 2. • At κ = 10%, chain 1 mostly consists of middle ages, while chain 2 of 10s, 20s, 60s, and 70s. • At κ = 50%, there is no such strong age-dependent tendency. But, for some combinations of gender, ethnic group, and age band, it is not equal division. For example, for Asian females, a majority of 40s are assigned to chain 1 but a majority of 50s and 60s are assigned to chain 2. The unsupervised algorithm is designed to divide instances into multiple clusters when gender and ethnic group information is unavailable. As shown in Appendix C.3, different κ's yield various clustering . Surprisingly, these different clusters still outperform the supervised algorithm. For example, at κ = 10%, let us consider the age band of 20s and 30s. If the references in chain 2 are used to estimate the ages of people in chain 1, the average error is 4.6 years. On the contrary, if the references in chain 1 are used for chain 2, the average error is −5.4 years. These opposite biases mean that people in chain 1 tend to look older than those in chain 2. These'looking-older' people in 20s and 30s compose the blue cluster (chain 1) together with most people in 40s and 50s in Figure 5. In this case,'looking-older' people in 20s and 30s are separated from'looking-younger' ones by the unsupervised algorithm. This is more effective than the gender-based or ethnic-group-based division of the supervised algorithm. Appendix C presents more on age estimation. Order learning was proposed in this work. In order learning, classes form an ordered set, and each class represents object instances of the same rank. Its goal is to determine the order graph of classes and classify a test instance into one of the classes. To this end, we designed the pairwise comparator to learn ordering relationships between instances. We then decided the class of an instance by comparing it with reference instances in the same chain and maximizing the consistency among the comparison . For age estimation, it was shown that the proposed algorithm yields the stateof-the-art performance even in the case of the single-chain hypothesis. The performance is further improved when the order graph is divided into multiple disjoint chains. In this paper, we assumed that the order graph is composed of disjoint chains. However, there are more complicated graphs, e.g. Figure 1 (a), than disjoint chains. For example, it is hard to recognize an infant's sex from its facial image . But, after puberty, male and female take divergent paths. This can be reflected by an order graph, which consists of two chains sharing common nodes up to a certain age. It is an open problem to generalize order learning to find an optimal order graph, which is not restricted to disjoint chains. During the chain membership update in Algorithm 1, we assign an instance x to chain k to maximize β k (x) subject to the regularization constraint. As mentioned in Section 4.1, in age estimation, this regularization is enforced for each age. Let X denote the set of θ-year-olds for a certain θ. Also, let K = {0, 1, . . ., K − 1} be the set of chains. Suppose that we should assign at least a certain number (L) of instances in X to each chain. This is done by calling RegularAssign(K, X, L) in Algorithm 2, which is a recursive function. Algorithm 2 yields the membership function c(x) as output. For example, c(x) = 1 means that x belongs to chain 1. Input: K = set of chains, X = set of instances, and L = minimum number 1: for each k ∈ K do Initialize chains 2: X k = ∅ 3: end for 4: for each x ∈ X do Irregular partitioning 5: c(x) = arg max k∈K β k (x) 6: X c(x) = X c(x) ∪ {x} 7: end for 8: km = arg min k∈K |X k | Chain of the minimum size 9: if |X km | ≥ L then 10: return 11: else 12: while |X km | < L do Increase X km 14: x = maxx∈X β km (x) 15: X km = X km ∪ {x} 17: end while 18: B TWO-STEP ESTIMATION There are 5 reference images for each age within range in this work. Thus, for the age estimation of a test image using the MC rule in, the test image should be compared with M = 330 reference images. However, we reduce the number of comparisons using a two-step approach. First, the test image is compared with the 35 references of ages 15, 25,..., 75 only, and a rough age estimateθ 1 is obtained using the MC rule. Second, it is compared with the 105 references of all ages within [θ 1 − 10,θ 1 + 10], and the final estimateθ 2 is obtained. Since there are at least 10 common references in the first and second steps, the two-step estimation requires at most 130 comparisons. • Setting A: 5,492 images of Europeans are randomly selected and then divided into training and testing sets with ratio 8:2 . • Setting B: About 21,000 images are randomly selected, while restricting the ratio between Africans and Europeans to 1:1 and that between females and males to 1:3. They are divided into three subsets (S1, S2, S3). The training and testing are done under two subsettings . -(B1) training on S1, testing on S2 + S3 -(B2) training on S2, testing on S1 + S3 • Setting C (SE): The entire dataset is randomly split into five folds, subject to the constraint that the same person's images should belong to only one fold, and the 5-fold crossvalidation is performed. • Setting D (RS): The entire dataset is randomly split into five folds without any constraint, and the 5-fold cross-validation is performed. Table 5 is an extended version of Table 2. It includes the of more conventional algorithms. We assess the proposed age estimator (1CH) on the FG-NET database . FG-NET is a relatively small dataset, composed of 1,002 facial images of 82 subjects. Ages range from 0 to 69. For FG-NET, the leave one person out (LOPO) approach is often used for evaluation. In other words, to perform tests on each subject, an estimator is trained using the remaining 81 subjects. Then, the are averaged over all 82 subjects. In order to assess the generalization performance, we do not retrain the comparator on the FG-NET data. Instead, we fix the comparator trained on the balanced dataset and just select references from the remaining subjects' faces in each LOPO test. For the comparator, the arithmetic scheme in∼ is tested as well as the default geometric scheme in∼. For comparison, MV is tested, but it is trained for each LOPO test., the proposed algorithm outperforms MV, even though the comparator is not retrained. These indicate that the comparator generalizes well to unseen data, as long as the training images cover a desired age range. Also, note that the geometric scheme provides better performances than the arithmetic scheme. Figure 6 compares MAEs according to a test age. Again, within the covered range, the proposed algorithm significantly outperforms MV especially when test subjects are older than 45. The ordering relationship between two instances can be categorized via the arithmetic scheme in∼ using a threshold τ or the geometric scheme in∼ using a threshold τ age. Table 8 lists the performances of the proposed algorithm (1CH) according to these thresholds. We see that the geometric scheme outperforms the arithmetic scheme in general. The best performance is achieved with τ age = 0.1, which is used in all experiments in the main paper. Note that the scores are poorer than those in Table 3, since the comparator is trained for a smaller number of epochs to facilitate this test. At τ age = 0.1, two teenagers are declared to be not'similar to' each other if their age difference is larger than about 1. Also, two forties are not'similar' if the age difference is larger than about 5. C.5 PERFORMANCE ACCORDING TO NUMBER OF REFERENCES Table 9: The performances of the proposed algorithm (supervised) on the balanced dataset according to the number of references for each age class (M/N). In general, the performances get better with more references. However, the performances are not very sensitive to M/N. They saturate when M/N ≥ 5. Therefore, we set M/N = 5 in this work. Figure 8: All reference images in the supervised 6CH. For some ages in certain chains, the balanced dataset includes less than 5 faces. In such cases, there are less than 5 references.

Coupling semi-supervised learning with self-supervised learning and explicitly modeling the self-supervised task conditioned on the semi-supervised one

We study whether and how adaptive data augmentation and knowledge distillation can be leveraged simultaneously in a synergistic manner for better training student networks.


Question: Represent the scientific article to find a short TLDR

Question: Ranking is a central task in machine learning and information retrieval. In this task, it is especially important to present the user with a slate of items that is appealing as a whole. This in turn requires taking into account interactions between items, since intuitively, placing an item on the slate affects the decision of which other items should be chosen alongside it. In this work, we propose a sequence-to-sequence model for ranking called seq2slate. At each step, the model predicts the next item to place on the slate given the items already chosen. The recurrent nature of the model allows complex dependencies between items to be captured directly in a flexible and scalable way. We show how to learn the model end-to-end from weak supervision in the form of easily obtained click-through data. We further demonstrate the usefulness of our approach in experiments on standard ranking benchmarks as well as in a real-world recommendation system. Ranking a set of candidate items is a central task in machine learning and information retrieval. Many existing ranking systems are based on pointwise estimators, where the model assigns a score to each item in a candidate set and the ing slate is obtained by sorting the list according to item scores ). Such models are usually trained from click-through data to optimize an appropriate loss function BID17. This simple approach is computationally attractive as it only requires a sort operation over the candidate set at test (or serving) time, and can therefore scale to large problems. On the other hand, in terms of modeling, pointwise rankers cannot easily express dependencies between ranked items. In particular, the score of an item (e.g., its probability of being clicked) often depends on the other items in the slate and their joint placement. Such interactions between items can be especially dominant in the common case where display area is limited or when strong position bias is present, so that only a few highly ranked items get the user's attention. In this case it may be preferable, for example, to present a diverse set of items at the top positions of the slate in order to cover a wider range of user interests. A significant amount of work on learning-to-rank does consider interactions between ranked items when training the model. In pairwise approaches a classifier is trained to determine which item should be ranked first within a pair of items (e.g., BID13 BID17 BID6 . Similarly, in listwise approaches the loss depends on the full permutation of items (e.g., BID7 BID47 . Although these losses consider inter-item dependencies, the ranking function itself is pointwise, so at inference time the model still assigns a score to each item which does not depend on scores of other items. There has been some work on trying to capture interactions between items in the ranking scores themselves (e.g., BID29 BID22 BID49 BID32 BID8 . Such approaches can, for example, encourage a pair of items to appear next to (or far from) each other in the ing ranking. Approaches of this type often assume that the relationship between items takes a simple form (e.g., submodular) in order to obtain tractable inference and learning algorithms. Unfortunately, this comes at the expense of the model's expressive power. In this paper, we present a general, scalable approach to ranking, which naturally accounts for high-order interactions. In particular, we apply a sequence-to-sequence (seq2seq) model BID35 to the ranking task, where the input is the list of candidate items and the output is the ing ordering. Since the output sequence corresponds to ranked items on the slate, we call this model sequence-to-slate (seq2slate). The order in which the input is processed can significantly affect the performance of such models BID39. For this reason, we often assume the availability of a base (or "production") ranker with which the input sequence is ordered (e.g., a simple pointwise method that ignores the interactions we seek to model), and view the output of our model as a re-ranking of the items. To address the seq2seq problem, we build on the recent success of recurrent neural networks (RNNs) in a wide range of applications (e.g., BID35 . This allows us to use a deep model to capture rich dependencies between ranked items, while keeping the computational cost of inference manageable. More specifically, we use pointer networks, which are seq2seq models with an attention mechanism for pointing at positions in the input BID38 . We show how to train the network end-to-end to directly optimize several commonly used ranking measures. To this end, we adapt RNN training to use weak supervision in the form of click-through data obtained from logs, instead of relying on ground-truth rankings, which are much more expensive to obtain. Finally, we demonstrate the usefulness of the proposed approach in a number of learning-to-rank benchmarks and in a large-scale, real-world recommendeation system. The ranking problem is that of computing a ranking of a set of items (or ordered list or slate) given some query or context. We formalize the problem as follows. Assume a set of n items, each represented by a feature vector x i ∈ R m (which may depend on a query or context). Let π ∈ Π denote a permutation of the items, where each π j ∈ {1, . . ., n} denotes the index of the item in position j. Our goal is to predict the output ranking π given the input items x. For instance, given a specific user query, we might want to return an ordered set of music recommendations from a set of candidates that maximizes some measure of user engagement (e.g., number of tracks played). In the seq2seq framework, the probability of an output permutation, or slate, given the inputs is expressed as a product of conditional probabilities according to the chain rule: DISPLAYFORM0 This expression is completely general and does not make any conditional independence assumptions. In our case, the conditional p(π j |π <j, x) ∈ ∆ n (a point in the n-dimensional simplex) models the probability of any item being placed at the j'th position in the ranking given the items already placed at previous positions. Therefore, this conditional exactly captures all high-order dependencies between items in the ranked list, including those due to diversity, similarity or other interactions. Our setting is somewhat different than a standard seq2seq setting in that the output vocabulary is not fixed. In particular, the same index (position) is populated by different items in different instances (queries). Indeed, the vocabulary size n itself may vary per instance in the common case where the number of items to rank can change. This is precisely the problem addressed by pointer networks, which we review next. We employ the pointer-network architecture of BID38 to model the conditional p(π j |π <j, x). A pointer network uses non-parametric softmax modules, akin to the attention mechanism of BID1, and learns to point to items in its input sequence rather than predicting an index from a fixed-sized vocabulary. Our seq2slate model, illustrated in FIG0, consists of two recurrent neural networks (RNNs): an encoder and a decoder, both of which use Long Short-term Memory (LSTM) cells BID14. At each encoding step i ≤ n, the encoder RNN reads the input vector x i and outputs a d-dimensional vector e i, thus transforming the input sequence {x i} n i=1 into a sequence of latent memory states {e i} n i=1. At each decoding step j, the decoder RNN outputs a d-dimensional vector d j which is used as a query in our attention function. The attention function takes as input the query d j ∈ R d and the set of latent memory states computed by the encoder {e i} n i=1 and produces a probability distribution over the next item to include in the output sequence as follows: DISPLAYFORM0 where W enc, W dec ∈ R d×d and v ∈ R d are learned parameters in our network, denoted collectively by parameter vector θ. The probability p DISPLAYFORM1, is obtained via a softmax over the remaining items and represents the degree to which the model points to input i at decoding step j. To output a permutation, the p j i are set to 0 for items i that already appear in the slate. Once the next item π j is selected, typically greedily or by sampling (see below), its embedding x πj is fed as input to the next decoder step. The input of the first decoder step is a learned d-dimensional vector, denoted as go in FIG0. Importantly, p θ (π|x) is differentiable for ant fixed permutation π which allows gradient-based learning (see Section 3). We note the following. (i) The model makes no explicit assumptions about the type of interactions between items. If the learned conditional in Eq. is close to the true conditional in Eq., then the model can capture rich interactions-including diversity, similarity or others. We demonstrate this flexibility in our experiments (Section 4). (ii) x can represent either raw inputs or embeddings thereof, which can be learned together with the sequence model. (iii) The computational cost of inference, dominated by the sequential decoding procedure, is O(n 2), which is standard in seq2seq models with attention. We also consider a computationally cheaper single-step decoder with linear cost O(n), which outputs a single vector p 1, from which we obtain π by sorting the values (similarly to pointwise ranking). We now turn to the task of training the seq2slate model from data. A typical approach to learning in ranking systems is to run an existing ranker "in the wild" and log click-through data, which are then used to train an improved ranking model. This type of training data is relatively inexpensive to obtain, in contrast to human-curated labels such as relevance scores, ratings, or rankings BID17. Formally, each training example consists of a sequence of items {x 1, . . ., x n} and binary labels (y 1, . . ., y n), with y i ∈ {0, 1}, representing user feedback (e.g., click/no-click). Our approach easily extends to more informative feedback, such as the level of user engagement with the chosen item (e.g., time spent), but to simplify the presentation we focus on the binary case. Our goal is to learn the parameters θ of p θ (π j |π <j, x) (Eq.) such that permutations π corresponding to "good" rankings are assigned high probabilities. Various performance measures R(π, y) can be used to evaluate the quality of a permutation π given the labels y, for example, mean average precision (MAP), precision at k, or normalized discounted cumulative gain at k (NDCG@k). Generally speaking, permutations where the positive labels rank higher are considered better. In the standard seq2seq setting, models are trained to maximize the likelihood of a target sequence of tokens given the input, which can be done by maximizing the likelihood of each target token given the previous target tokens using Eq.. During training, the model is typically fed the ground-truth tokens as inputs to the next prediction step, an approach known as teacher forcing BID43. Unfortunately, this approach cannot be applied in our setting since we only have access to weak supervision in the form of labels y (e.g clicks), rather than ground-truth permutations. Instead, we show how the seq2slate model can be trained directly from the labels y. One potential approach, which has been applied successfully in related tasks BID3 BID48, is to use reinforcement learning (RL) to directly optimize for the ranking measure R(π, y). In this setup, the objective is to maximize the expected ranking metric obtained by sequences sampled from our model: E π∼p θ (.|x) [R(π, y)]. One can use policy gradients and stochastic gradient ascent to optimize θ. The gradient is formulated using the popular REINFORCE update BID42 and can be approximated via Monte-Carlo sampling as follows: DISPLAYFORM0 where k indexes ranking instances in a batch of size B, π k are permutations drawn from the model p θ and b(x) denotes a baseline function that estimates the expected rewards to reduce the variance of the gradients. RL, however, is known to be a challenging optimization problem and can suffer from sample inefficiency and difficult credit assignment. As an alternative, we propose supervised learning using the labels y. In particular, rather than waiting until the end of the output sequence (as in RL), we wish to give feedback to the model at each decoder step. Consider the first step, and recall that the model assigns a score s i to each item in the input. We define a per-step loss (s, y) which essentially acts as a multi-label classification loss with labels y as ground truth. Two natural, simple choices for are cross-entropy loss and hinge loss: DISPLAYFORM0 hinge (s, y) = max{0, 1 − min i:yi=1 DISPLAYFORM1 whereŷ i = y i / j y j, and p i is a softmax of s, similar to Eq.. Intuitively, with cross-entropy loss we try to assign high probabilities to positive labels (see also BID20, while hinge loss is minimized when scores of items with positive labels are higher than scores of those with negative labels. Notice that both losses are convex functions of the scores s. To improve convergence, we consider a smooth version of the hinge-loss where the maximum and minimum are replaced by their smooth counterparts: smooth-max(s; γ) = 1 γ log i e γsi (and smooth minimum is defined similarly, using min i (s i) = − max i (−s i)).If we simply apply a per-step loss from Eq. to all steps of the output sequence while reusing the labels y at each step, then the loss is invariant to the actual output permutations (e.g., predicting a positive item at the beginning of the sequence has the same cost as predicting it at the end). Instead, we let the loss at each decoding step j depend on the items already chosen, so no further loss is incurred after a label is predicted correctly. In particular, for a fixed permutation π, define the sequence loss: DISPLAYFORM2 where S = {s j} n j=1, and π<j (s j, y) depends only on the indices in s j and y which are not in the prefix permutation π <j = (π 1, . . ., π j−1) (see Eq. FORMULA4). Including a per-step weight w j can encourage better performance earlier in the sequence (e.g., w j = 1/ log(j + 1)). Furthermore, if optimizing for a particular slate size k is desired, one can restrict this loss to just the first k output steps. Since teacher-forcing is not an option, we resort to feeding the model its own previous predictions, as in; BID31. In this case, the permutation π is not fixed, but rather depends on the scores S. Specifically, we consider two policies for producing a permutation during training, sampling and greedy decoding, and introduce their corresponding losses. The greedy policy consists of selecting the item that maximizes p θ (·|π <j, x) at every time step j. The ing permutation π * then satisfies π * j = argmax i p θ (π j = i|π * <j) and our loss becomes L π *. The greedy policy loss is not continuous everywhere since a small change in the scores s may in a jump between permutations, and therefore L π. Specifically, the loss is non-differentiable when any s j has multiple maximizing arguments. Outside this measure-zero subspace, the loss is continuous (almost everywhere), and the gradient is well-defined. Sampling policy The sampling policy consists of drawing each π j from p θ (·|π <j, x). The corresponding loss E[L] = π p θ (π)L π (θ) is differentiable everywhere since both p θ (π) and L π (θ) are differentiable for any permutation π (See appendix for a direct derivation of E[L] as a function of S). In this case, the gradient is formulated as: DISPLAYFORM0 which can be approximated by: DISPLAYFORM1 where b(x k) is a baseline that approximates L π k (θ). Applying stochastic gradient descent intuitively decreases both the loss of any sample (right term) but also the probability of drawing samples with high losses (left term). Notice that our gradient calculation differs from scheduled sampling which instead computes the loss of the sampled sequences (right term) but ignores the probability of sampling high loss sequences (left term). We found it helpful to include both terms, which may apply more generally to training of sequence-to-sequence models BID11. For both training policies, we minimize the loss via stochastic gradient descent over mini-batches in an end-to-end fashion. We evaluate the performance of our seq2slate model on a collection of ranking tasks. In Section 4.1 we use learning-to-rank benchmark data to study the behavior of the model. We then apply our approach to a large-scale commercial recommendation system and report the in Section 4.2. Implementation Details We set hyperparameters of our model to values inspired by the literature. All experiments use mini-batches of 128 training examples and LSTM cells with 128 hidden units. We train our models with the Adam optimizer BID19 and an initial learning rate of 0.0003 decayed every 1000 steps by a factor of 0.96. Network parameters are initialized uniformly at random in [−0.1, 0.1]. To improve generalization, we regularize the model by using dropout with probability of dropping p dropout = 0.1 and L2 regularization with a penalty coefficient λ = 0.0003. Unless specified otherwise, all use supervised training with cross-entropy loss xent and the sampling policy. At inference time, we report metrics for the greedy policy. We use an exponential moving average with a decay rate of 0.99 as the baseline b(x) in Eq. FORMULA3 and Eq.. When training the seq2slate model with REINFORCE, we use R = NDGC@10 as the reward function and do not regularize the model. We also considered a bidirectional encoder RNN BID34 but found that it did not lead to significant improvements in our experiments. To understand the behavior of the proposed model, we conduct experiments using two learning-torank datasets. We use two of the largest publicly available benchmarks: the Yahoo Learning to Rank Challenge data (set 1), 1 and the Web30k dataset. Table 1: Performance of seq2slate and other baselines on data generated with diverse-clicks. We adapt the procedure proposed by BID18 to generate click data. The original procedure is as follows: first, a base ranker is trained from the raw data. We select this base ranker by training all models in the RankLib package, 3 and selecting the one with the best performance on each data set (MART for Yahoo and LambdaMART for Web30k). We generate an item ranking using the base model, which is then used to generate training data by simulating a user "cascade" model: a user observes each item with decaying probability 1/i η, where i is the base rank of the item and η is a parameter of the generative model. This simulates a noisy sequential scan. An observed item is clicked if its ground-truth relevance score is above a threshold (relevant: {2, 3, 4}, irrelevant: {0, 1}), otherwise no click is generated. To introduce high-order interactions, we augment the above procedure as follows, creating a generative process dubbed diverse-clicks. When observing a relevant item, the user will only click if it is not too similar to previously clicked items (i.e, diverse enough), thus reducing the total number of clicks. Similarity is defined as being in the smallest q percentile (i.e., q = 0.5 is the median) of Euclidean distances between pairs of feature vectors within the same ranking instance: d ij = x i − x j. We use η = 0 (no decay, since clicks are sparse anyway due to the diversity term) and q = 0.5. This modification to the generative model is essential for our purpose as the original data does not contain explicit inter-item dependencies. We also discuss variations of this model below. Using the generated training data, we train both our seq2slate model and baseline rankers from the RankLib package: AdaRank BID46, Coordinate Ascent BID24, LambdaMART BID45, ListNet BID7, MART BID10, Random Forests BID5, RankBoost BID9, RankNet BID6. Some of these baselines use deep neural networks (e.g., RankNet, ListNet), so they are strong state-ofthe-art models with comparable complexity to seq2slate. The in Table 1 show that seq2slate significantly outperforms all the baselines, suggesting that it can better capture and exploit the dependencies between items in the data. To better understand the behavior of the model, we visualize the probabilities of the attention from Eq. for one of the test instances in Fig. 2. Interestingly, the model produces slates that are close to the input ranking, but with some items demoted to lower positions, presumably due to the interactions with previous items. We next consider several variations of the generative model and of the seq2slate model itself. Results are reported in TAB2. The rank-gain metric per example is computed by summing the positions change of all positive labels in the re-ranking, and this is averaged over all examples (queries). TAB2, we compare the different training variants outlined in Section 3, namely cross entropy with the greedy or sampling policy, a smooth hinge loss with γ = 1.0, and REINFORCE. We find that supervised learning with cross entropy generally performs best, with the smooth hinge loss doing slightly worse. Our weakly supervised training methods have positive rank gain on all datasets, meaning they improve over the base ranker. Results from TAB2 in the appendix) suggest that training with REINFORCE yields comparable on Yahoo but significantly worse on the more challenging Web30k dataset. We find no significant difference in performance between relying on the greedy and sampling policies during training. Table 3: Performance compared to a competitive base production ranker on real data. One-step decoding We compare seq2slate to the model which uses a single decoding step, referred to as one-step decoder (see Section 2). In TAB2 we see that this model has comparable performance to the sequential decoder. This suggests that when inference time is crucial, as in many real-world systems, one might prefer the faster single-shot option. One possible explanation for the comparable performance of the one-step decoder is that the interactions in our generated data are rather simple and can be effectively learned by the encoder. By contrast, in Section 4.2 we show that on more complex real-world data, sequential decoding can perform significantly better. Sensitivity to input order Previous work suggests that the performance of seq2seq models are often sensitive to the order in which the input is processed BID39 BID26. To test this we consider the use of seq2slate without relying on the base ranker to order the input, but instead items are fed to the model in random order. The in TAB2 (see shuffled data) show that the performance is indeed significantly worse in this case, which is consistent with previous studies. It suggests that reranking is an easier task than ranking from scratch. Adaptivity to the type of interaction To demonstrate the flexibility of seq2slate, we generate data using a variant of the diverse-clicks model above. In the similar-clicks model, the user also clicks on observed irrelevant items if they are similar to previously clicked items (increasing the number of total clicks). As above, we use the pairwise distances in feature space d ij to determine similarity. For this model we use q = 0.5, and η = 0.3 for Web30k, η = 0.1 for Yahoo, to keep the proportion of positive labels similar. The in the appendix (see TAB4) show that seq2slate has comparable performance to the baseline rankers, with slightly better performance on the harder Web30k data. This demonstrates that our model can adapt to various types of interactions in the data. We also apply seq2slate to a ranking problem from a large-scale commercial recommendation system. We train the model using massive click-through logs (comprising roughly O(10 7) instances) with cross-entropy loss, the greedy policy, L2-regularization and dropout. The data has item sets of varying size, with an average n of 10.24 items per example. We learn embeddings of the raw inputs as part of training. Table 3 shows the performance of seq2slate and the one-step decoder compared to the production base ranker on test data (of roughly the same size as the training data). Significant gains are observed in all performance metrics, with sequential decoding outperforming the one-step decoder. This suggests that sequential decoding may more faithfully capture complex dependencies between the items. Finally, we let the learned seq2slate model run in a live experiment (A/B testing). We compute the click-through rate (CTR) in each position (#clicks/#examples) for seq2slate. The production base ranker serves traffic outside the experiment, and we compute CTR per position for this traffic as well. Fig. 3 shows the difference in CTR per position, indicating that seq2slate has significantly higher CTR in the top positions. This suggests that seq2slate indeed places items that are likely to be chosen higher in the ranking. In this section we discuss additional related work. Our work builds on the recent impressive success of seq2seq models in complex prediction tasks, including machine translation BID35 BID1, parsing BID37, combinatorial optimization BID38 BID3, multi-label classification BID41 BID26, and others. Our work differs in that we explicitly target the ranking task, which requires a novel approach to training seq2seq models from weak feedback (click-through data). Most of the work on ranking mentioned above uses shallow representations. However, in recent years deep models have been used for information retrieval, focusing on embedding queries, documents and query-document pairs BID15 BID12 BID27 BID40 BID28 ) (see also recent survey by BID25). Rather than embedding individual items, in seq2slate a representation of the entire slate of items is learned and encoded in the RNN state. Moreover, learning the embeddings (x) can be easily incorporated into the training of the sequence model to optimize both simultaneously end-to-end. Closest to ours is the recent work of BID0, where an RNN is used to encode a set of items for re-ranking. Their approach uses a single decoding step with attention, similar to our one-step decoder. In contrast, we use sequential decoding, which we find crucial in certain applications (see Section 4.2). Another important difference is that their training formulation assumes availability of full rankings or relevance scores, while we focus on learning from cheap click-through data. Finally, Santa BID33 recently proposed an elegant framework for learning permutations based on the so called Sinkhorn operator. Their approach uses a continuous relaxation of permutation matrices (i.e., the set of doubly-stochastic matrices). Later, BID23 combined this with a Gumbel softmax distribution to enable efficient learning. However, this approach is focused on reconstruction of scrambled objects, and it is not obvious how to extend it to our ranking setting, where no ground-truth permutation is available. We presented a novel seq2slate approach to ranking sets of items. We found the formalism of pointer-networks particularly suitable for this setting. We addressed the challenge of training the model from weak user feedback to improve the ranking quality. Our experiments show that the proposed approach is highly scalable and can deliver significant improvements in ranking . Our work can be extended in several directions. In terms of architecture, we aim to explore the Transformer network BID36 in place of the RNN. Several variants can potentially improve the performance of our model, including beam-search inference BID44, and training with Actor-Critic BID2 or SeaRNN BID21 ) and it will be interesting to study their performance in the ranking setting. Finally, an interesting future work direction will be to study off-policy correction BID16 Since the terms are continuous (and smooth) in S for all j and π <j, so is the entire function.

Represent the TLDR for finding the corresponding scientific article

A pointer network architecture for re-ranking items, learned from click-through logs.

Represent the short summary for finding the corresponding scientific article (long)

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.