Question: Represent the article for finding a single-sentence TLDR:

Question: We propose a novel adversarial learning framework in this work. Existing adversarial learning methods involve two separate networks, i.e., the structured prediction models and the discriminative models, in the training. The information captured by discriminative models complements that in the structured prediction models, but few existing researches have studied on utilizing such information to improve structured prediction models at the inference stage. In this work, we propose to refine the predictions of structured prediction models by effectively integrating discriminative models into the prediction. Discriminative models are treated as energy-based models. Similar to the adversarial learning, discriminative models are trained to estimate scores which measure the quality of predicted outputs, while structured prediction models are trained to predict contrastive outputs with maximal energy scores. In this way, the gradient vanishing problem is ameliorated, and thus we are able to perform inference by following the ascent gradient directions of discriminative models to refine structured prediction models. The proposed method is able to handle a range of tasks, \emph{e.g.}, multi-label classification and image segmentation. Empirical on these two tasks validate the effectiveness of our learning method. This work focuses on applying adversarial learning BID9 to solve structured prediction tasks, e.g., multi-label classification and image segmentation. Adversarial learning can be formalized as a minimax two-player game between structured prediction models and discriminative models. Discriminative models are learned to distinguish between the outputs predicted by the structured prediction models and the training data, while structured prediction models are learned to predict outputs to fool discriminative models. Though structured prediction models are trained by the gradients of discriminative models, existing methods rarely use discriminative models to improve structured prediction models at the inference stage. A straightforward way of utilizing discriminative models for inference is to follow the ascent gradient directions of discriminative models to refine the predicted outputs. However, due to the wellknown gradient vanishing problems, the gradients of discriminative models are almost zero for all the predicted outputs. It is difficult to resolve the gradient vanishing problems, because they are caused by the training instability of the existing adversarial learning framework. Consequently, most existing methods do not use the information from discriminative models to refine structured prediction models. Most existing adversarial learning methods take discriminative models as classifiers. If discriminative models well separate real and predicted samples, they tend to assign the same scores to all predicted samples. Differently, energy-based models BID21 BID12 usually predict different energy scores for different samples. Therefore, we propose to train discriminative models as energy-based models to ameliorate the gradient vanishing problems. In our framework, discriminative models are learned to assign scores to evaluate the quality of predicted outputs. Structured prediction models are learned to predict outputs that are judged to have maximum scores by discriminative models. In this way, discriminative models are trained to approximate continues value functions which evaluate the quality of predicted output. The gradients of discriminative models are not zero for predicted outputs. Thus, we can refine structured prediction models by following the ascent gradient directions of discriminative models at the inference stage. In this paper, we refer our method as learning discriminative models to refine structured prediction models (LDRSP).The proposed method learns discriminative models utilizing the data generated by the structured prediction models. BID12 found that the key to learning deep value networks is generating proper training data. We propose to augment the training set of discriminative models by following the data generation methods proposed in BID12. At the training stage, we simultaneously run the inference algorithm to generate extra training samples utilizing models in previous iterations. These samples are useful since they are generated along the gradient-based inference trajectory utilized at the inference stage. We also augment the training set with adversarial samples BID10. These samples are used as negative samples to train the discriminative models. To validate our method, experiments are conducted on multi-label classification, binary image segmentation, and 3-class face segmentation tasks, and experimental indicate that our method can learn discriminative models to effectively refine structured prediction models. This work has two contributions: We propose a novel adversarial learning framework for structured prediction, in which the information captured by discriminative models can be used to improve structured prediction models at the inference stage. We propose to learn discriminative models to approximate continues value functions that evaluate the quality of the predicted outputs, and thus ameliorate the gradient vanishing problems. Recently, adversarial learning has been well studied on producing high-resolution and high-quality images BID8 BID18, improving the training stability and getting rid of problems like model collapse BID11 BID29 BID31, image to image translation BID14, semantic image segmentation BID27 BID15 and neural dialogue generation. These methods learn discriminative models and generative models in an adversarial way, but discriminative models are abandoned once models are trained. Since discriminative models are learned to capture the discriminative distributions between the generated samples and the training data, we argue that they are able to be utilized to make the generated samples more real. As we know, we are the first to learn discriminative models that can be directly utilized to refine structured prediction models. There are some researches that utilize iterative inference methods. Since Denoising Autoencoders (DAEs) can estimate the gradients of energy distribution functions, BID30 propose to learn DAEs to capture probability distributions of realistic images. In the inference process, DAEs can be utilized to iteratively refine generated images. BID32 expand similar ideas to the image segmentation. Different from these methods, in which feed-forward networks are utilized to iteratively refine the generated samples, we adopt a gradient-based inference to find samples with the zero task loss. Learning discriminative models as energy-based models has been recently studied BID35 BID7. These works only constrain that discriminative models predict lower energy scores for the training data than the generated samples. However, our method learns discriminative models to approximate value functions that evaluate the quality of the generated samples. Moreover, we propose to generate extra training samples to learn discriminative models, so that discriminative models can better capture the probability distributions of the sample space. There is a rising interest in energy-based structured prediction BID36 BID6 BID33. BID1 proposed to add constraints to the neural network parameters such that the output of the neural network is a convex function of (some of) the inputs. BID3 introduced Structured Prediction Energy Network (SPEN). SPEN relies on a max-margin surrogate objective to ensure that the neural network predicts the lowest energy value for the ground-truth label. BID4 improved SPEN by proposing an end-to-end version of SPEN, which directly back-propagates through a computation graph that unrolls gradient-based energy minimization. Inspired by the reinforcement learning, BID12 proposed a Deep Value Network (DVN) that directly learns to evaluate the quality of different output configures. Compared with these methods, our method adversarially learns energy-based models and structured prediction models rather than learns energy-based models alone. We propose a novel adversarial learning framework for structured prediction, in which discriminative models D(x, y; θ d) can be used to refine structured prediction models G(x; θ g) at the inference stage. In our method, discriminators are treated as energy-based models, which take both input objects x and possible outputs y as inputs and predict scores in the range of. We assume that at the training stage, one can get access to an oracle value function v * (y, y *), which evaluates the quality of y corresponding to x. Here y * is the ground-truth label. The discriminators are learned to mimic the behavior of the oracle value functions. Following BID12, we utilize intersection over union (IOU) and F 1 metrics as the oracle value functions for image segmentation and multi-label classification, respectively, which are defined on (y, y DISPLAYFORM0 DISPLAYFORM1 Here y ∩ y * denotes the number of dimension i where both y i and y * i are active and y ∪ y * denotes the number of dimensions where at least one of y i and y * i is active. y i and y * i denote the i-th variable of y and y * . To apply v * (y, y *) to the continuous output y, the notions of intersection and union are extended by using element-wise min and max operators, DISPLAYFORM2 DISPLAYFORM3 We propose to learn discriminators D(x, y; θ d) to estimate v * (y, y *). The learning of discriminators can be understood from a regression setting with z = (x, y) as inputs and v = v * (x, y) as the target outputs. The structured prediction models are learned to predict outputs scored highly by discriminators. Discriminators and structured prediction models are respectively learned by optimizing: DISPLAYFORM4 DISPLAYFORM5 Here DISPLAYFORM6, and L g is a surrogate loss. Equation 5 can be understood in two aspects: It's a modified version of least-square GAN BID28 loss, and discriminators are learned to predict oracle values v * g for the samples predicted by structured prediction models; It learns an energy-based model using training samples that consist of ground-truth samples (x, y *, 1) and predicted samples (x, G(x; θ g), v * g ). The second term of Equation 6 regularizes structured prediction models such that the predicted samples tend to have higher scores. Once models are trained, discriminators can refine structured prediction models by utilizing a gradient-based inference. The outputs predicted by structured prediction models are updated following the ascent gradient directions of discriminators that lead to the high scores: DISPLAYFORM7 DISPLAYFORM8 Here, P Y denotes an operator that projects the predicted outputs back to the feasible set of solutions. In a simple case, where Y = M, the P Y operator clips the predicted outputs. It is observed in experiments, i.e., the learned discriminators tend to give small gradients during the gradient-based inference. One reason is that the predicted outputs of structured prediction models are already close to the ground-truth labels. In order to further improve the predicted outputs by using the gradientbased inference and overcome the small-gradient issue, we use the normalized gradient method, i.e., DISPLAYFORM9 Gygli et al. FORMULA0 proposed to simultaneously generate training samples to learn their energy-based model, i.e., Deep Value Network (DVN) in the training process. We follow their method to generate extra training samples to learn discriminators. The training samples are a set of tuples (input x, output y, oracle value v DISPLAYFORM10 Here N is the size of the training set, and x (i), y (i) and v * (i) respectively denote the i-th image, i-th output and i-th oracle value in the training set. Similar to BID12, we utilize two different methods to generate training samples:• Inference samples. At each training iteration, structured prediction models are first used to predict samples. Then, we take these samples as initial solutions and run a gradient-based inference to find high-valued samples of discriminators. These samples are useful since they are generated along the inference trajectory at the inference stage. These samples can be generated during training by using models in a previous iteration BID12.• Adversarial samples. Maximize the loss: DISPLAYFORM11 2 with respect to y using a gradient-based optimizer BID10. These samples serve as negative samples to learn discriminators. Similar to the inference samples, the adversarial samples are also generated during training. To utilize these training samples to learn discriminators, we add another loss term to the Equation 5, and the new objective function is as follows: DISPLAYFORM12 We utilize the Adam optimizer BID20 with the momentum term β 1 = 0.5 to train structure prediction models and discriminators. At each training iteration, we randomly generate inference or adversarial samples and update the parameters of the structured prediction models and the discriminators according to Equation and Equation. The models are trained until convergence. Experiments are conducted on three tasks: multi-label classification, binary image segmentation, and 3-class face segmentation. We compare our LDRSP to other state-of-the-art adversarial learning methods on these tasks, and are reported in Section 4.1, 4.2 and 4.3. The code is implemented using the deep learning framework, i.e., Tensorflow BID0. We use standard benchmarks of this task, namely Bibtex and Bookmarks introduced by BID19 in this section. On these two datasets, tags need to be predicted for text inputs and multiple labels are possible for each input. A two-layer neural network BID3 ) is utilized as our structured prediction model. The same network architecture of DVN is adopted as our discriminator, which consists of one or two hidden layers with Softplus non-linearities. Following BID3, a cross-entropy loss is used as the surrogate loss. Besides the proposed LDRSP, different adversarial learning methods, e.g., GAN BID9, WGAN+GP BID11, LSGAN BID28 and EBGAN BID35 are utilized to learn the structured prediction models and the discriminators. These structured prediction models and discriminators have the same network architecture. For all the methods, the hyper-parameter exploration is performed, and we follow the gradient directions of the discriminators to refine the structured prediction models at the inference stage. The experimental are reported in TAB0. For all the adversarial learning methods, we report both the prediction of structured prediction models G and the prediction refined by the discriminators D. As it shows, the proposed LDRSP successfully learns D that can be utilized to refine the G and achieve state-of-the-art performance on both Bibtex and Bookmarks datasets. The refinement improves the performance of G by 3.6 % on Bibtex dataset and 1.1 % on Bookmarks dataset. However, for other adversarial learning methods, the refinement leads to negligible improvements or even decrements in performance. For example, the refinement of EBGAN improves the performance by 0.2 % on Bookmarks dataset, and the refinement of LSGAN decreases the performance by 0.1 % on Bibtex dataset. Following BID3, we implement a baseline model by training structured prediction models via minimizing a cross-entropy loss, i.e., the first term of Equation FORMULA5. Comparing the performance of the baseline model and the performance of our adversarially learned structured prediction models, we notice that jointly learning structured prediction models and discriminators via the proposed method greatly improves the performance of structured prediction models. The performance of state-of-the-art structured prediction methods e.g., logistic regression, a twolayer neural network learned with a cross-entropy loss, SPEN BID3, PRLR , and DVN BID12 ) is also reported in TAB0. Although our method, the DVN, and the SPEN have the same energy network architecture, our method outperforms the DVN and the SPEN on both Bibtex and Bookmarks datasets. Our method also greatly improves over feed-forward models: the logistic regression, the two-layer neural network, and the PRLP. BID24 37.2 30.7 NN baseline BID3 38.9 33.8 SPEN BID3 42.2 34.4 PRLR BID24 44.2 34.9 DVN BID12 44.7 37.1 SPM baseline 38.9 32.8 SPM (GAN) BID9 38.7 32.7 GAN BID9 38.6 32.8 SPM (LSGAN) BID28 40.0 32.4 LSGAN BID28 39.9 32.5 SPM (WGAN + GP) BID11 39.6 30.5 WGAN + GP BID11 39.7 30.6 SPM (EBGAN) BID35 40.2 32.3 EBGAN BID35 41.5 32.5 SPM (Our LDRSP) 42.8 37.2 LDRSP (Our)46.4 38.3 Compared with the multi-label classification task, the image segmentation task is more challenging due to the high-dimensional output space and the complex correlation among variables in the segmentation masks. We utilize the Weizmann horses dataset BID5 in this section. It is a commonly used dataset for binary image segmentation which consists of 328 left oriented horse images and their corresponding binary segmentation masks. Following BID12 BID23, all images and segmentation masks are resized to 32 × 32. The segmentation of horses at this low resolution is challenging and requires models to capture strong priors of the horse shape, since some thin parts of the horse like legs, tails are almost invisible in the images. We follow the experimental protocol of BID23 to split the Weizmann horses dataset and report on the same testing set. In the experiment, we adopt the fully convolutional network (FCN) BID26 baseline model proposed in BID12 as our structured prediction model. It consists of three 5 × 5 convolutional layers and two deconvolution layers. We find that using discriminators similar to PatchGAN discriminators BID25 improves the prediction performance. Therefore, our discriminators are designed to map (x, y) to score matrices. Here x ∈ R W ×3, y ∈ R W ×C, W is the number of pixels in an image, and, C is the number of object classes. We view image segmentation as pixel-level multi-label classification. Instead of approximating discriminators to the intersection over union (IOU) metric, we calculate the F 1 metric for each pixel and estimate an oracle value function v DISPLAYFORM0 where y i ∈ R C denotes the i-th row of y. Our discriminators are implemented by modifying the FCN baseline model: the number of object classes is set to 1 and a sigmoid function is added at the end of the discriminator to ensure v(x, y) ∈W. Similar to the multi-label classification, a cross-entropy loss is adopted as the surrogate loss. For both discriminators and FCNs, we adopt Adam optimizers and set the learning rates to 0.01. Data augmentation is utilized by using random 24 × 24 patches cropped from the 32 × 32 images and by randomly mirroring the images. We empirically find that setting the inference step size η to be 4.0 and setting the number of inference steps to be 30 achieve the best performance. At the inference stage, we divide each test image into 36 crops. The proposed method is utilized to estimate segmentation masks for these crops. The estimated segmentation masks are averaged to obtain the final segmentation mask. We notice that the proposed method usually converges within 20 inference steps. Table 2: The comparison of IOU between our LDRSP and other adversarial learning methods on the Weizmann horses dataset. FCN is adopted as our structured prediction model. Mean IOU % Global IOU % CHOPPS BID23 69.9 -DVN BID12 84.1 84.0 FCN baseline BID12 78.56 78.7 FCN (GAN) BID9 79.8 79.7 GAN BID9 79.7 79.5 FCN (LSGAN) BID28 80.3 79.9 LSGAN BID28 79.7 79.3 FCN (WGAN + GP) BID11 80.6 80.4 WGAN + GP BID11 80.5 80.4 FCN (EBGAN) BID35 80.9 80.7 EBGAN BID35 80.9 80.8 FCN (Our LDRSP) 81.3 81.3 LDRSP (Our) 85.5 85.4As commonly done in the literature, we report the mean image IOU as well as the IOU over the whole testing set on the Weizmann horses dataset in Table 2. A higher IOU score means a more accurate segmentation mask. It is clear that the proposed LDRSP outperforms other state-of-the-art methods on both metrics. It can be observed that using the discriminators of the LDRSP to refine the FCN improves the performance by 4.2 % on the Mean IOU metric and 4.1 % on the Global IOU metric. It indicates that our LDRSP learns discriminators that are able to estimate stronger horse shape priors than the FCN. As it can be seen, the refinement of discriminators learned by other adversarial learning methods, e.g., GAN BID9, WGAN+GP BID11, LSGAN BID28 decreases the performance. It's due to the fact that these methods learn discriminators as classifiers which assign almost the same scores to all the predicted samples. Thus, it's difficult to utilize these discriminators to improve the performance of FCNs. Our LDRSP outperforms the DVN BID12. It indicates that utilizing the proposed method to jointly learn energy-based models and structured prediction models advances over learning energy-based models alone. The qualitative on the Weizmann horses dataset are shown in FIG0. It can be seen that FCN shows poor performance in segmenting thin parts like legs and generating single-connected segmentation masks. The proposed method utilizes discriminators to refine the predictions of the FCN by filling the missing part (e.g., FIG0, seventh and eighth row, far left images), generating legs to connect disconnected parts (e.g., FIG0, seventh and eighth row, first and second images from the right). We utilize the Labeled Faces on the Wild (LFW) dataset BID13 to evaluate our method on 3-class face segmentation. This dataset contains more than 13,000 images, which were first introduced for face recognition and later were annotated on a subset of 2,927 images for face segmentation. The annotations provide superpixel-level labels which consist of three classes: face, hair, and . Since our method generates pixel-level labels, we map pixel-level labels to superpixel-level labels by using the most frequent labels in a superpixel as the superpixel's label following BID34 BID12. We adopt the training, validation, and testing splits proposed in BID17 BID34 BID12. The network architecture and the data augmentation are the same as those utilized on the Weizmann horses dataset. BID12 92.44 FCN baseline BID12 95.36 FCN (GAN) BID9 95.53 GAN BID9 95.54 FCN (LSGAN) BID28 95.51 LSGAN BID28 95.52 FCN (WGAN + GP) BID11 95.59 WGAN + GP BID11 95.59 FCN (EBGAN) BID35 95.50 EBGAN BID35 95 We compare the proposed method with other methods on the LFW dataset in TAB1. It can be observed that the FCN baseline outperforms the DVN BID12 by 2.92 % on the superpixel accuracy metric, while our method outperforms the FCN baseline by 1.11 %. The performance improvement between our method and the DVN BID12 on the LFW dataset is more significant than the performance improvement on the Weizmann horses dataset. It indicates that utilizing the proposed method to jointly learn an energy-based model and a FCN greatly improves the performance of the energy-based model, when the output space is large. The qualitative of the LDRSP on the LFW dataset are shown in FIG1. As it can be observed, our method can generate high-quality hair and face segmentation masks that are close to the ground-truth labels. This paper proposes a novel learning framework, in which discriminative models are learned to refine structured prediction models. Discriminative models are trained as energy-based models to estimate scores that measure the quality of generated samples. Structured prediction models are trained to predict contrastive samples with maximum energy scores. Once models are learned, we perform inference by following the ascent gradient directions of discriminative models to refine structured prediction models. We apply the proposed method to multi-label classification and image segmentation tasks. The experimental indicate that discriminative models learned by the proposed methods can effectively refine generative models. As the future work, we will explore different ways to generate extra training samples and apply our method to more challenging tasks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a novel adversarial learning framework for structured prediction, in which discriminative models can be used to refine structured prediction models at the inference stage.

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We propose RaPP, a new methodology for novelty detection by utilizing hidden space activation values obtained from a deep autoencoder. Precisely, RaPP compares input and its autoencoder reconstruction not only in the input space but also in the hidden spaces. We show that if we feed a reconstructed input to the same autoencoder again, its activated values in a hidden space are equivalent to the corresponding reconstruction in that hidden space given the original input. In order to aggregate the hidden space activation values, we propose two metrics, which enhance the novelty detection performance. Through extensive experiments using diverse datasets, we validate that RaPP improves novelty detection performances of autoencoder-based approaches. Besides, we show that RaPP outperforms recent novelty detection methods evaluated on popular benchmarks. How can we characterize novelty when only normality information is given? Novelty detection is the mechanism to decide whether a data sample is an outlier with respect to the training data. This mechanism is especially useful in situations where a proportion of detection targets is inherently small. Examples are fraudulent transaction detection , intrusion detection , video surveillance (; b), medical diagnosis and equipment failure detection (; ;). Recently, deep autoencoders and their variants have shown outstanding performances in finding compact representations from complex data, and the reconstruction error has been chosen as a popular metric for detecting novelty . However, this approach has a limitation of measuring reconstruction quality only in an input space, which does not fully utilize hierarchical representations in hidden spaces identified by the deep autoencoder. In this paper, we propose RAPP, a new method of detecting novelty samples exploiting hidden activation values in addition to the input values and their autoencoder reconstruction values. While ordinary reconstruction-based methods carry out novelty detection by comparing differences between input data before the input layer and reconstructed data at the output layer, RAPP extends these comparisons to hidden spaces. We first collect a set of hidden activation values by feeding the original input to the autoencoder. Subsequently, we feed the autoencoder reconstructed input to the autoencoder to calculate another set of activation values in the hidden layers. This procedure does not need additional training of the autoencoder. In turn, we quantify the novelty of the input by aggregating these two sets of hidden activation values. To this end, we devise two metrics. The first metric measures the total amount of reconstruction errors in input and hidden spaces. The second metric normalizes the reconstruction errors before summing up. Note that RAPP falls back to the ordinary reconstruction-based method if we only aggregate input values before the input layer and the reconstructed values at the output layer. Also, we explain the motivations that facilitated the development of RAPP. We show that activation values in a hidden space obtained by feeding a reconstructed input to the autoencoder are equivalent to the corresponding reconstruction in that hidden space for the original input. We refer the latter quantity as a hidden reconstruction of the input. Note that this is a natural extension of the reconstruction to the hidden space. Unfortunately, we cannot directly compute the hidden reconstruction as in the computation of the ordinary reconstruction because the autoencoder does not impose any correspondence between encoding-decoding pairs of hidden layers during the training. Nevertheless, we show that it can be computed by feeding a reconstructed input to the autoencoder again. Consequently, RAPP incorporates hidden reconstruction errors as well as the ordinary reconstruction error in detecting novelty. With extensive experiments, we demonstrate using diverse datasets that our method effectively improves autoencoder-based novelty detection methods. In addition, we show by evaluating on popular benchmark datasets that RAPP outperforms competing methods recently developed. Our contributions are summarized as follows. • We propose a new novelty detection method by utilizing hidden activation values of an input and its autoencoder reconstruction, and provide aggregation functions for them to quantify novelty of the input. • We provide motivation that RAPP extends the reconstruction concept in the input space into the hidden spaces. Precisely, we show that hidden activation values of a reconstructed input are equivalent to the corresponding hidden reconstruction of the original input. • We demonstrate that RAPP improves autoencoder-based novelty detection methods in diverse datasets. Moreover, we validate that RAPP outperforms recent novelty detection methods on popular benchmark datasets. Various novelty detection methods with deep neural networks rely on the reconstruction error (; ;), because discriminative learning schemes are not suitable for highly class-imbalanced data which is common in practice. Unsupervised and semi-supervised learning approaches handle such imbalance by focusing on the characterization of normality and detecting samples out of the normality. Variational Autoencoders (VAE) were reported to outperform vanilla autoencoders for novelty detection based on reconstruction error . To carry out the novelty detection outlined in this approach, an autoencoder needs to be trained only with normal data. The autoencoder encodes the training data, which comprises of only normal data in this case, into a lower-dimensional space and decodes them to the input space. To test novelty, an input value is fed to the autoencoder to produce a reconstructed value and calculate the distance between the input and reconstructed values. This distance is the reconstruction error. A higher reconstruction error means that the input value cannot be encoded onto the lower-dimensional space that represents normal data. Therefore, the input value can be marked as a novelty if its reconstruction error exceeds a certain threshold. Instead of autoencoders, Generative Adversarial Networks (GAN) have been also suggested to model a distribution of normal data . Despite the same purpose of discovering a simpler, lower-dimensional representation, the training criterion for GAN is focusing on the quality of data generation rather than the reconstruction quality of training data. Recently, several pieces of research have combined autoencoders and adversarial learning to meet both criteria in dimension reduction and data generation (; ;). One limitation of these methods based on the ordinary reconstruction error is that they do not exploit all the information available along the projection pathway of deep autoencoders. We will explain how to leverage this information for novelty detection in the next section. From the viewpoint of the diversity and ratio of the normal data in novelty detection, there are two cases available. The first case is when a small fraction of classes are normal. This case has been studied in a one-class classification context, and usually evaluated by organizing training data into a collection of samples belonging to a small number of normal classes (; ; ;). The second case is when a majority of classes are assigned as normal (; ; ;). In this case, normal data is more diverse, and the training data is consist of samples of a relatively large number of normal classes: e.g., nine digits of MNIST. One setup does not dominate the other, but depending on applications, either can be more suitable than the other. Different methods may perform differently in both cases. In this paper, we evaluate RAPP and other competing methods with experiments in both setups. In this section, we describe the proposed novelty detection method RAPP based on an autoencoder. The main idea is to compare hidden activations of an input and its hidden reconstructions along the projection pathway of the autoencoder. To be precise, we project the input and its autoencoder reconstruction onto the hidden spaces to obtain pairs of activation values, and aggregate them to quantify the novelty of the input. For the aggregation, we present two metrics to measure the total amount of difference within each pair. An autoencoder A is a neural network consisting of an encoder g and a decoder f, responsible for dimension reduction and its inverse mapping to the original input space, respectively: i.e. A = f • g. For the purpose, training the autoencoder aims to minimize difference between its input x and output A(x). The space that the encoder g constitutes is called the latent space, and provides more concise representation for data than the input space. Due to this unsupervised representation learning property, the autoencoder has been widely used for novelty detection. Specifically, training an autoencoder on normal data samples, novelty of a test sample x is measured by the following reconstruction error: The test sample x is more likely to be novel as the error (x) becomes larger, because it means that x is farther from the manifold that the autoencoder describes. Although this approach has shown promising in novelty detection, the reconstruction error alone does not fully exploit information provided by a trained autoencoder especially when its architecture is deep. In other words, hierarchical information identified by the deep architecture is being ignored. This is rather unfortunate because hierarchical representation learning is one of the most successfully proven capabilities of deep neural networks. To fully leverage that capability, below we will describe the way to exploit hidden spaces to capture the difference between normal and novel samples in more detail. Let A = f • g be a trained autoencoder where g and f are an encoder and a decoder, and be the number of hidden layers of g. Namely, g = g • · · · • g 1. We define partial computation of g as follows: Let x be an input vector, andx be its reconstruction by A: i.e.,x = A(x). In addition to comparing x andx in the input space, as the ordinary approach does, we examine them in hidden spaces along a projection pathway of A. More precisely, feeding x andx into A, we obtain pairs (h i,ĥ i) of their hidden representations where Figure 1a illustrates the procedure of computing h i andĥ i. As a , novelty of the sample x is quantified by aggregating The overall procedure of RAPP is summarized in Algorithm 1. To clearly state the required variables to construct H, we write the algorithm with the for loop in Lines 3-5, but in practice, all of them can be computed by feed-forwarding one time each of x andx to g. Note that RAPP is indeed a generalization of the ordinary reconstruction method with defining g 0 as the identity function and s ord as follows. The quantity that RAPP computes, the hidden activation of the reconstruction input, is equivalent to the hidden reconstruction of the input. Sincef = f, computingĥ 2 =ĥ 2 does not require explicitly evaluatingf i but only g i and f =f. Algorithm 1: RAPP to compute a novelty score. Input: Sample x, trained autoencoder A = f • g, the number of layers, and aggregation s. Output: Novelty score S. In this paper, we provide two metrics s SAP and s N AP which more extensively utilize H than s ord. Those are especially suited when no prior knowledge exists for the selection of layers to derive a novelty metric, which commonly happens when modeling with deep neural networks. Note that, however, more elaborate metrics can be designed if we have knowledge on or can characterize the spaces. This is the most straightforward metric that one can define on H. For a data sample x, SAP is defined by summing the square of Euclidean distances for all pairs in H: Although SAP is intuitive, it does not consider properties of hidden spaces; distance distributions of pairs in H may be different depending on the individual hidden spaces. For instance, the magnitude of distances can depend on layers, or there may exist correlated neurons even across layers which are unintentionally emphasized in SAP. To capture clearer patterns, we propose to normalize the distances via two steps: orthogonalization and scaling.; given a training set X, let D be a matrix whose i-th row corresponds to d(x i) for x i ∈ X, andD be the column-wise centered matrix of D. For the normalization, we computeD = U ΣV, SVD ofD, to obtain its singular values Σ and right singular vectors V. For a given data sample x, we define s N AP as follows: where µ X is the column-wise mean of D, and d(x) is expressed as a column vector. Note that s N AP is equal to the Mahalanobis distance with the covariance matrix V ΣΣV. Although SVD computation time is quadratic in the number of columns of the target matrix, we observe that its impact is relatively small in practical setups. See Appendix A for more details. One natural question in using the ordinary reconstruction method is as follows: why do we investigate only the input space? Or, why do we not use information in hidden spaces? While the reconstruction error in the input space is extensively employed, any similar concept does not exist in hidden spaces. One reason is that the corresponding encoding and decoding layers are not guaranteed to express the same space: e.g. permuted dimensions. This is because the autoencoder objective does not have any term involving activations from intermediate hidden layers. As a , f:i+1 (g(x)) cannot be considered a reconstruction of g:i (x), except for i = 0 with which they become the ordinary reconstruction of and input to an autoencoder, respectively. Nevertheless, in this section, we will show that there is an indirect way to compute the hidden reconstruction. Precisely, we will show thatĥ i (x) = g:i (A(x)) is indeed equivalent to a reconstruction of g:i (x). The overall mechanism is depicted in Figure 1b. Let A = f • g be a trained autoencoder, and M 0 = {A(x): x ∈ R n } be the low dimensional manifold that A describes : i.e., ∀x ∈ M 0, x = A(x). Defining M i = {g :i (x): x ∈ M 0 }, which is the low dimensional image of M 0 defined by g:i, g and f restricted on M 0 and M, respectively, are inverse functions of each other. Quantifying Hidden Reconstruction We first assume that there exists a decoderf The second condition makesf:i+1 a proper decoder corresponding to g i+1:, and thus,f enables to define the i-th hidden reconstructionĥ i (x) as follows: Finally, we conclude thatĥ i (x) is equal toĥ i (x) for x ∈ M 0 as follows. where we do not needf i for computingĥ i (x), but only g i and f. Note that for x ∈ M 0 already on the manifold, its i-th hidden reconstructionĥ i (x) becomes equal to its corresponding hidden input. For x / ∈ M 0, its hidden reconstructionĥ i (x) will differ from the input h i (x). Existence off Since x = A(x) for x ∈ M 0, g i and f i are one-to-one functions from M i−1 and M i, respectively. Let us definef i = g for M i; then it also holdsf = g −1 for M. This implies x = (f • g)(x) for x ∈ M 0, and consequently,f = f on M. This definition off i satisfies the two conditions above, and as discussed, we are able to compute hidden reconstructions given an input x, through computing the i-th hidden activation of the reconstructed input: i.e.ĥ i (x) = (Existence off with Neural Networks Given g i, if the symmetric architecture forf i is used, we may not be able to learnf i = g −1 i . Neural networks are, however, highly flexible frameworks in which we can deal with models of arbitrary function forms by adjusting network architecture. This property enables us to design a layer capable of representingf i . For instance, even iff i is too complicated to be represented with a single fully connected layer, we can still approximatef i by stacking multiple layers. Hence, given g i,f i can be represented by neural networks. In this section, we evaluate RAPP in comparison to existing methods. To this end, we tested the methods on several benchmarks and diverse datasets collected from Kaggle and the UCI repository which are suitable for evaluating novelty detection methods. The datasets from Kaggle and the UCI repository are chosen from problem sets of anomaly detection and multi-class classification, summarized in Table 1 . We note that MI-F and MI-V share the same feature matrix, but are considered to be different datasets because their labels normal and abnormal are assigned by different columns: i.e. machine completed and pass visual inspection, respectively. We use these datasets to compare RAPP with standard autoencoder-based methods described in Section 5.2. To compare RAPP with novelty detection methods in recent literatures, we also use popular benchmark datasets for evaluating deep learning techniques: MNIST and F-MNIST . For theses datasets, we do not take pre-split training and test sets, but instead merge them for post-processing. Novelty detection detects novel patterns by focusing on deviations from model-learned normal patterns. Thus, training sets contain only normal samples and test sets contain both normal and anomaly samples in our evaluation setups. Precisely, if a dataset contains an anomaly label, we assign all samples with that label to the test set for detection. If a dataset does not have any anomaly labels, we consider the following two setups. • Multimodal Normality: A single class is chosen to be the novelty class and the remaining classes are assigned as the normal class. This setup is repeated to produce sub-datasets with all possible novelty assignments. For instance, MNIST in a set of datasets with 10 different novelty classes. • Unimodal Normality: In contrast to the multimodal normality setup, we take one class for normality, and the others for novelty. For instance, MNIST in a set of datasets with 10 different normal classes. We applied these two setups to STL, OTTO, SNSR, MNIST, and F-MNIST datasets. We compare RAPP and the other methods using Area Under Receiver Operating Characteristic (AUROC). Note that we do not employ thresholding-based metrics such as F 1 score because access to abnormal samples is only allowed in testing time. Hence, we focus on the separability of models for novelty with AUROC. For the datasets in Table 1, we compare the effectiveness of the reconstruction error, SAP and NAP for three models: Autoencoder (AE), Variational Autoencoder (VAE), Adversarial Autoencoder (AAE) . For the benchmark datasets, recent approaches including OCNN , GPND , DSVDD and GT are available. To obtain the performances of the existing approaches, we downloaded their codes and applied against our problem setups. Given novelty classes, we create the test sets by randomly selecting samples while maintaining novelty ratios to 35% for the multimodal and 50% for the unimodal normality setups, respectively. Note that the expectation value of AUROC is invariant to the novelty ratio. We use symmetric architecture with fully-connected layers for the three base models, AE, VAE, and AAE. Each encoder and decoder has 10 layers with different bottleneck size. For the Kaggle and UCI datasets, we carry out PCA for each dataset first. The minimum number of principal components that explain at least 90% of the variance is selected as the bottleneck size of the autoencoders. We set bottleneck size to 20 for benchmark datasets. Leaky-ReLU (a) activation and batch normalization layers are appended to all layers except the last layer. We train AE, VAE and AAE with Adam optimizer , and select the model with the lowest validation loss as the best model. For training stability of VAE, 10 Monte Carlo samples were averaged in the reparamterization trick to obtain reconstruction from the decoder. In the calculation of SAP and NAP, we excluded reconstructions in the input space for MNIST and F-MNIST. Each AUROC score is obtained by averaging AUROC scores from five trials to reduce the random errors in training neural networks. More are provided in Appendix: standard deviations in Appendix B, comparison to baselines other than autoencoder variants C, and the effect of varying hidden layers involved in RAPP computation in Appendix D. Table 2 summarizes the of our performance evaluation; the best score for each model is in bold, and the best score for each dataset with an underline. Since STL, OTTO, SNSR, MNIST, and F-MNIST do not have anomaly labels, their scores are averaged over all possible anomaly class assignments. For instance, the AUROC value for OTTO in the unimodal normality setup is the average of 9 AUROC values with different anomaly class assignments. In Table 2, RAPP shows the highest AUROC scores for most of the cases. If we examine the performance for each dataset, RAPP achieves the best for 13 cases out of 15 (see the underlines). Table 3 summarizes the comparison of RAPP to recent novelty detection methods. As in Table 2, AUROC values are calculated by averaging from 10 cases with different anomaly class assignments for both datasets. Except for the unimodal F-MNIST setup, NAP outperforms all competing methods regardless of base model choice. Notably, NAP combined with VAE always shows the best performance, which is even higher than that of GT relying on image-specific data transformations for all cases. In this paper, we propose a novelty detection method which utilizes hidden reconstructions along a projection pathway of deep autoencoders. To this end, we extend the concept of reconstruction in the input space to hidden spaces found by an autoencoder and present a tractable way to compute the hidden reconstructions, which requires neither modifying nor retraining the autoencoder. Our experimental show that the proposed method outperforms other competing methods in terms of AUROC for diverse datasets including popular benchmarks. A SVD COMPUTATION TIME We compare running times of training an autoencoder and computing SVD for NAP. We choose two packages for the SVD computation: Pytorch SVD and fbpca provided in https://fbpca. readthedocs.io/en/latest/. Since the time complexity of SVD is linear in the number of data samples 1, we mainly focus on the performance of SVD with varying the number of columns of the input matrix that SVD is applied. To obtain variable sizes of the columns, we vary the depth and bottleneck size of autoencoders. The is shown below. Notably, Pytorch SVD utilizing GPU is at least 47x faster than training neural networks. Even, fbpca running only on CPU achieves at least 2.4x speedup. The detailed setups to obtain the matrices for the experiment are given in the 1 20 100 20 40 20 90 2 18 100 18 40 18 90 3 16 100 16 40 16 90 4 14 100 14 40 14 90 5 12 100 12 40 12 90 6 10 100 10 40 10 90 7 8 100 8 40 8 90 8 6 100 6 40 6 90 9 4 100 4 40 4 90 10 2 100 2 40 2 90 11 2 80 2 30 2 70 12 2 60 2 20 2 50 13 2 40 2 10 2 30 14 2 20 2 10 B STANDARD DEVIATIONS OF EXPERIMENTAL We provide the standard deviations of the in We investigate the performance of NAP while increasing the number of hidden layers involved in the NAP computation. Specifically, we consider two ways for the increment: 1) adding hidden layers one by one from the input layer (forward addition), and 2) adding hidden layers one by one from the bottleneck layer (backward addition). Experimental on two datasets are shown below. For most cases, more hidden layers tend to in higher performance. The values are obtained from one trial, not averaged over 5 trials as done in Section 5.

Represent the TLDR for finding the corresponding scientific article.

A new methodology for novelty detection by utilizing hidden space activation values obtained from a deep autoencoder.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Learning preferences of users over plan traces can be a challenging task given a large number of features and limited queries that we can ask a single user. Additionally, the preference function itself can be quite convoluted and non-linear. Our approach uses feature-directed active learning to gather the necessary information about plan trace preferences. This data is used to train a simple feedforward neural network to learn preferences over the sequential data. We evaluate the impact of active learning on the number of traces that are needed to train a model that is accurate and interpretable. This evaluation is done by comparing the aforementioned feedforward network to a more complex neural network model that uses LSTMs and is trained with a larger dataset without active learning. When we have a human-in-the-loop during planning, learning that person's preferences over plan traces becomes an important problem. These preferences can be used to choose a plan from amongst a set of plans that are comparable by the planner's cost metrics. Such a plan would naturally be more desired by the human. The user may not like to constantly dictate their preferences, and may not always be in the loop during execution. Thus, it is important for the user's preference function to be learned well, and for the user to be able to verify them. For verification, there ought to be a way to interpret how the model's decisions were made, and verify how faithful the learned model is to the user's preferences. A user's preferences function may be quite complex with dependencies over different subsets of features. The utility of some features maybe non-linear as well. Such a preference function may require a fair amount of information to approximate. We cannot expect a single user to give feedback over a large set of traces to get the relevant information. So Active learning, with a sufficiently expressive user interface for feedback, is essential to minimize queries and redundant information. In this work, our objective was to model the user's preferences over plan traces. There do exist techniques that efficiently represent and reason about preference relationships. CP-nets BID1 and Generalized additive independence BID2 ) models are typically used to represent preferences over sets of variables without consideration to the order in which they appear. While these models can be adapted to handle sequential data, they are not intended for it. LTL rules, however, can capture trajectory preferences very well and are used in PDDL 3.0 BID3, and LPP BID0. However, it can be very hard for a user to express their preferences in this form. We discuss existing approaches in more detail and the differences with respect to our work under the related work section. In our approach to learning preferences, we want to efficiently identify the relevant features and the degree to which they affect the preference score of a plan. We thus employ a feature-directed active learning approach that specifically picks plan traces that are most informative about the feature's effects on preference. After active learning, we encode a plan trace in terms of the relevant features it contains. We gather a set of training data from active learning, along with the user's preference score to help train a simple Neural Network (NN) that we call the FeatureNN model. We use a Neural Network as they can approximate complex functions to a good degree. Our approach is in one way, related to Generalized Additive Independence in that we try to learn a utility function over pertinent features, but we do not explicitly define or restrict the form of any utility functions. Rather a simple one hidden-layer feed-forward neural network learns the functions, dependencies, and relative weights over the relevant features. The FeatureNN then predicts a preference score for each plan reflecting the user's preferences. We also compare the performance of the FeatureNN to another SequenceNN model that processes sequential data using an LSTM BID6 module. The SequenceNN is not trained with data from active learning, but with a larger dataset of traces with ratings. This is to evaluate how efficient our active learning approach is with respect to the number of traces. Specifically, we compare the number of traces required by SequenceNN and FeatureNN for the same accuracy and interpretability. Neural networks, unlike linear functions, are not as easy to interpret. Even simple NN with a single hidden layer can be a challenge. We help the user interpret the decisions of the neural network by showing how the preference score is affected by removing different features of a plan trace. This is similar to using Saliency Maps BID7 in images to explain what parts of the image contributed to the classification. In this way, we can explain to the user what plan features contributed to the preference value and by how much. The difference in preference score should correspond to the user's expectations as per their preference model. The more similar the effect of changes are to the user's preferences, the more interpretable the NN model is to the user as it approximates well their own preference function. Such a method of explaining a decision(score) is also related to explaining using counterfactuals BID5. Here the counterfactual is the plan trace without a specific feature. Additionally, when the specific features used to compute preferences comes from the user's feedback (during active learning), this interpretability is obviously improved. We present our work by first defining the problem before delving into the methodology of our approach. In the Methodology section, we discuss the domain used, the user preference model, and the feature-directed active learning process. We also discuss the two neural network models used to learn the preference model, viz. the FeatureNN and the SequenceNN models. Then we present our experimental in which we compare the two models with respect to their accuracy in predicting the preference score, as well as interpretability. Lastly, we discuss the and possible extensions to the work. Given a Domain D, with a set of features F, and a planner P, the problem is to learn the preference function F p that captures the user's preference model U p and scores traces accordingly. The types of preferences we learn in this work are a function of the feature set F of the domain and not hidden variables or action costs. The user U is available to rate a plan trace on its preference, and annotate what features contributed positively or negatively to the rating. Features can be categorical or cardinal(count), and involve sequences. Plans are rated between with higher values indicating a greater preference. If there are no features in the plan that either contributed positively or negatively to the preference, then the preference score is 0.5.An equivalent problem formulation assumes that instead of a domain D and planner P, we are given as input a large enough set of plan traces B (backlog of traces) over a relevant set of initial and goal states. We assume that this set of plans covers the space of possible preferences that the user might have. For our experiments, we chose to use gridworld with features that any human can relate to. We chose gridworld as it is easy to quickly generate many diverse plans that cover the range of features. Given the domain and a task, we go through r rounds of active learning. Each round comprises of r t traces. Both r and r t are hyperparameters. For our experiments, we set the number of rounds at r = 3. After acquiring the data, we train the NN model and test it on a hidden set of traces. We now go over different parts of our methodology in detail. The objective in our gridworld domain, which we call Journey − W orld, is to travel from home to the campsite which is shown in the grid in FIG1. Each step of a plan corresponds to a cell of the grid. While some cells are empty, there a lot of cells that have features. These features can be eateries(like a coffee shop, restaurant), landmarks(natural history museum) or activities(visiting the library, watching a movie). The user can move through any of these states before reaching the campsite. A majority of the cells also contain landscapes like mountains, lake, sea or industries. The user is not allowed to move through Landscape cells. Moving through cells adjacent to Landscape cells corresponds to seeing the landscape along the journey. For example, if a step in the plan goes through a cell which is adjacent to a lake, this corresponds to the plan going through a state where the user passed by a lake. All non-landscape features (like coffee, donut) are binary features in a plan trace i.e. the user has either visited one or not. On the other hand, the landscape features are cardinal, i.e. we count the number of such landscape features in the plan trace. We assume that the count of cardinal features can make a difference in the preference score. In total, there are 13 features in Journey − W orld. We had designed Journey − W orld with simple and commonly understood features to make it easier for subsequent human-studies. We assume people will have preferences over these features. For our current experiments, we chose to use a completely defined user preference model to represent the user. This made it easier for us to test and debug our methodology. In future extensions of this work, we will include evaluations with human trials. The user's preference model is defined as follows DISPLAYFORM0 where n(x) = min(x, 2), C ∈ {0, 1}, D ∈ {0, 1} and L, I ∈ N C is a binary variable that is 1 when the plan trace has a coffee. D is also a binary variable and represents a Donut. CD represents a binary variable set to true when the plan trace a coffee and a donut. When CD is true, C and D are false and this dependency affects the preference score computation as shown in the preceding equation. L and I represent the number of lake and industry regions respectively. These are cardinal features, and the preference of the plan increases based on their count, up to 2, and then stops increasing. The function itself, while simple to understand, is non-linear and hidden underneath a large hypothesis space of functions that could be learned in the domain, over a larger set of features (13 in total for the Journey − W orld domain). In our experiments, we programmed a separate module to rate and annotate plan traces based on an input preference function like the one described previously. This synthetic human is what rates and annotates in the active learning process that we will describe shortly. Using a synthetic human helped speed up the testing and debugging process, and gives us a baseline noiseless scenario to test against. The current user interface FIG1 (a)) for Journey − W orld displays the entire grid. Icons are used to show the features present in cells. The plans for each round of active learning are then shown one at a time. The plan steps are visualized as a line going from the home to the campsite. The user has to input a rating to indicate their preference for the plan based on the features that are visited. They can also annotate features of the plan that they like(green) or dislike(red) as shown in FIG1 (b). The user can click on the N ext P lan button to then move on to the next plan. The interface automatically switches to the next round of Active Learning when the current round's plans have all been rated. In our active learning process, we go through multiple (r) rounds of feedback. Each successive round utilizes the knowledge from previous rounds to select the most informative queries. In the first round, the user is shown the most diverse set of plan trajectories that were generated for the domain. We choose diverse plans because in the first round we do not have any knowledge of what features might affect the user's preference and hence, we want to cover the feature space as much as possible in r t = 30 traces. In order to get the required diverse plan set, we first generate a large number of plan traces(10000 plans) over a user specified set of initial and goal states that we refer to as the backlog of plans, B. In our current experiments we only have one initial and one goal state. We are easily able to generate such a large backlog of plans because it is a type of gridworld domain. We did not want the computational cost of diverse plan generation to hamper the work. This is a computational cost that needs to be considered when working with other domains. The plans generated cover the entire feature space of the Journey − W orld domain. We then select the 30 most diverse plans within the set of backlog plans for the first round. We will now discuss how this is determined. The diversity score between any two plans p a and p b is denoted by d(p a, p b). The diversity is based on the sum of feature count differences for features f ∈ F that are present p a and p b. For a particular feature f, we compute the feature count difference f ∆. Rather than use the difference in count per feature, we use a geometric series sum as computed in Equation 1. The first count in the difference contributes 1, the second count contributes a 0.5, the third contributes 0.25 and so forth. So the count difference for a single feature contributes to at most 2 to the diversity computation. This avoids any single feature from dominating the diversity computation. The diversity between two traces is computed as the average f E∆ (Equation 2) over all the features in the domain. Finally, we calculate the backlog-diversity d B for a plan p using equation 3. The backlog-diversity is the average pairwise diversity over every other plan in the backlog. Using this diversity score, we select the top r t plans (r t = 30 in our experiments) for the user feedback. DISPLAYFORM0 After the first round of diverse plans, we then make use of the ratings and annotations provided by the user in the first round to generate the most informative plan traces for the subsequent rounds. Given our acquired knowledge of relevant features from the previous round, our objective now is to figure out the effects and dependencies between these features. We also want to select traces for the next round that are more likely to be rated either significantly higher or lower. This region of data is typically harder to get as we expect most data to be closer to the average. In order to estimate which plan traces would be either most preferred or least preferred, we use a fast weak predictor that predicts the rating of any arbitrary plan p given prior knowledge. We need the predictor to be fast as we have to give traces or queries for the next round in a short amount of time. The weak predictor estimates a value for each feature based on the prior annotated data. It can then estimate the score of an unrated plan as just the sum of the features present in it. The value of each feature is scored using a quick and simple method. First, for each scored plan trace p with rating r p, the feature f is given a score f p score for that plan by equation 4. Then the feature's score, f score, is computed as the average f p score over all plans that the feature appears in. Then to predict the score for an unrated plan, the weak predictor assigns a score predict(p) which is the sum of f score for all features present in the plan. DISPLAYFORM1 In addition wanting plans that are likely to be rated much higher or much lower, we also want the next round traces to have two more properties. We still want to include some diversity in the plan traces with respect to the overall backlog of traces to uncover features that we might have missed in the first round. Additionally, we want to maintain some similarity in traces between the rounds. We think that the similarity between plans reduces the cognitive load on the user as they need not parse wholly different traces. Given a plan p, we denote its similarity to the already scored traces as S(p).Finally, we assign a combined weighted score of p c to all the plans in the backlog given by equation 5. The top r t = 30 plans are then picked for the next round, and in this way the active learning proceeds for r rounds. For our experiments r is 3 rounds. DISPLAYFORM2 Preference Learning using Neural NetworksFor learning the preference function we used two models, SequenceNN model and FeatureNN model. The SequenceNN model uses an LSTM module in it. We considered an LSTM based model as they are well suited to learning patterns overs sequential data. The input plan trace was encoded such that each step was an encoding over the features of the cell visited at that step. There are 13 features in total, and so each step is a 13-dimensional vector. We do not provide or restrict the input to only the features that the user annotated during active learning for the SequenceNN model. We wanted to test how easily the model could still figure out the relevant features and learn the preference function well. The training data for the SequenceNN model was a set of rated plan traces. We varied the number of traces given from as small as 30 to 12,000 in increasingly larger step sizes. A plan trace would be a N × 13 array where N is the plan length. We trained the model for 10 epochs with a batch size of 8, a learning rate of 0.01 and using stochastic gradient descent. The SequenceNN module in our model has 16 memory cells. After processing the plan trace through the LSTM module, we concatenate the output vector and memory nodes of the LSTM module and pass it through a single fully connected hidden layer, followed by the output layer which outputs the preference score between. The model summary is in FIG2. The idea is that the LSTM module output and memory, at the end of processing the sequence, will have the necessary information related to the sequence for predicting the score. For the FeatureNN model, the input was an encoding of the plan trace that only comprised of the features the user annotated as relevant during active learning. The entire plan trace was summarized into one encoded vector. For example, in the user preference model in our experiments, only 5 features matter to the user. We determine what these features are through active learning, and then defined our FeatureNN model accordingly to take a 5-dimensional vector as input. For example, if a plan trace had a step with coffee and steps that passed by 3 lakes, then the values at the corresponding indices are set to 1, and 3 respectively. Note that since coffee is a binary feature, even if two coffee steps were in the plan, it's value in the encoding is only either 1 or 0. As for the model description of FeatureNN, it was a simple fully connected neural net with one hidden layer of 4 dimensions and one output layer. The model summary is in FIG3. Note that 4 dimensions or nodes for the hidden layer is not a magic number, and would need to be larger if there were more features. We reduced the number of dimensions for the hidden layer until the were measurably worse. To train the FeatureNN model we vary the number of traces per round r t from 5 to 50 traces for r = 3 rounds. Since the dataset size is very small (smallest is 15 traces), we create 200 duplicates of the data points uniformly and train for 10 epochs. We also shuffle the data and train with a batch size of 8, a learning rate of 0.01, and using stochastic gradient descent. A user can interpret a Neural Network's behavior through analyzing what features are salient to its decision, and by how much. This can be analyzed by adding or removing features and seeing the ant effect on the predicted score. When done over a set of different traces, the user can intuit what mattered and how much. With this in mind, we compute a measure of interpretability we call the Attribution Error AE. The AE for a feature f of a Plan p is computed as follows: DISPLAYFORM0 where U p is the preference function of the user (true model of preferences) and F p is the learned preference function. AE is simply the difference in the effect of the feature on the preference scores. The overall AE for each test plan p, AE(p), is the average of AE(p, f) for all f present in p. We compute the AE score for the test set as the average over only the top 10% of AE(p) errors. We do this because neural networks can sometimes have enough capacity to memorize many cases and increase accuracy. So it can predict the correct preference score of both of the original trace and modified trace (with dropped feature) by the memory of very similar traces. It would then seem like it's preference function predicts the same way as the ground-truth preference function, but it maybe using unrelated features. Therefore, it is in the failure cases that we get a true measure of its generalization and how faithful it is to the true model of preferences. That is why we use the average over the top 10% of AE(p) errors. These failure cases could correspond to the cases when a rare or unseen pattern of features are input, and thus not memorized. When varying the number of training input traces given to the SequenceNN model, we observed that the accuracy improved (error decreased) as expected (Figure 4(a) ). Surprisingly, even with 30 traces, it was able to predict with an error of 2.5% over the test set of unseen 1000 traces. We attribute this to the fact that there are enough simple correlations with other features that can predict the score well for the preference function that we tested with. This is evidenced by the fact that the interpretability measure (Attribution Error) is very low for 30 traces(Figure 5 (a) ); The attribution error was greater than 0.3 and the value range of AE is. Additionally, we give the most diverse N traces for each training set size to the SequenceNN model. Diverse traces are more likely to contain relevant information. The interpretability of the LSTM model was not impressive. The attribution error did decrease over the range of training set sizes, but only as low as 0.09 as shown in Figure 5(a). Given that the preference scores are between, this would correspond to a 9 percent error after 7500 rated traces. Needless to say, it is unreasonable to expect a single human to rate 7500 traces. We also tried varying the size of the SequenceNN model from 16 to 64 dimensions. This improved accuracy by a minuscule amount (order of 1e−4), and interpretability did not improve. The performance of the FeatureNN model was significantly better both in accuracy(lower error) and interpretability than the SequenceNN model as seen in Figure 5. This should come as no surprise since we restrict the input space based on user feedback (knowledge) on relevant features. This also restricts the hypothesis space of functions that the simple feed-forward network could search over. We think this will make it more likely that the NN will find a good and faithful What is interesting to note is that the interpretability, as measured by the AE error drops to as little as 2.5 % in as little as 60 traces (20 traces per round over 3 rounds) for FeatureNN model as shown in Figure 5b. It drops below 2 % with 150 traces. The FeatureNN model with 90 traces is as accurate as the SequenceNN with 7500 traces in our problem, with 8% less Attribution Error (more interpretable). While we expected FeatureNN to be better, we did not expect such a large difference in efficiency. Even with as little as 13 features and a relatively uncomplicated preference function, a sufficiently powerful SequenceNN model did not find the underlying preference function. Instead, it found correlations that predicted the preference score to a very high level of accuracy. This, unfortunately, makes the model suffer in interpretability. As the number of features increases, the hypothesis space of a NN will increase significantly. This makes it much more likely for any NN to find spurious correlations, and suffer in interpretability. So active learning and using a simpler NN becomes very important for learning preferences in plan traces. As for prior feature knowledge, we assumed knowledge about what features were categorical (binary in our experiments) and what features were cardinal. Rather than assume this knowledge, we can get this from the user as well, and reduce the assumptions about the domain features. Alternatively, we could have just encoded all features as cardinal features, and let the neural network determine what features were categorical. While this is certainly possible, we think it better to get this knowledge from the user and encode the plan trace based on this knowledge. This makes the job of the neural network easier, and less likely to learn spurious correlations. In our current encoding of features in FeatureNN model and our experiments, we have not included a preference dependency that considers the number of steps between features. For example, I would like to have a donut within 3 plan steps after having a coffee. This omission was not intentional. One can easily encode such a sequential feature as a variable as well. The number of steps between the two (state) features becomes a cardinal variable to represents this sequential feature. Two well known paradigms for learning, representing and reasoning over preferences are CP-nets and Generalized additive independence (and their variants). Both of them were intended for preferences over outcomes. Each outcome can be comprised of many parts(decisions). One can think of each decision as choosing a value for a variable. The user would have preferences over the possible outcomes, or there maybe a utility (value) associated to each outcome. In CPnets BID1, the decisions or variables are represented in a graph, and there exists dependencies over variables. The preferences of a variable's values are affected by the value of the parent variables. The CP in CP-nets stands for Ceteris Paribus or "all else being equal". Here, the all else refers to the parents of the node, and when they are equal, then a particular set of preference orderings for the child variable's values hold. The knowledge of the dependence graph is either known apriori, or can be queried from the user. Once the hierarchy of dependence is known, the user is then queried about preferences at each node. For CPnets to be used in plan trace preferences, we would have to ask the user what the dependencies are. Then we would have to ask the user for their relative preferences over features, given fixed parent feature values. Note that the variables for plan preferences may also have to incorporate information about order. So there are significantly more variables (features) to consider in sequential data versus unordered data. We think querying for such knowledge is very demanding, and not natural for preferences over plan traces. We believe it is more natural for the user to specify the relevant conditional dependencies over features while annotating a plan trace. Additionally, we think it easier to give a preference value for the plan trace rather than relative preference orderings over the features in the domain. The features could include sequential dependencies or positiondependent features. We think it would be hard for the user to be able to describe sequential features and the relative ordering over them. Lastly, CP-nets do not compute utility values and some outcomes can be incomparable for a particular network of dependencies. In our problem, we would like a total order over the plans, to select the most preferred plan. So it helps to have a utility/preference value for every plan trace. On the other hand, GAI BID2 ) models do provide a single utility value for a set of features. As stated in their work, they provide an additive decomposition of a utility function (into sub-utility functions) in situations where single attributes are not additively independent, but (possibly overlapping) subsets of attributes are. Since the subsets of attributes for the different sub-utility functions can overlap, one must query either with only global queries or a combination of local queries (over the subsets of features) with global queries to calibrate (as was done in the GAI work) (Braziunas and Boutilier 2006)(p. 3). To learn a GAI with active learning from a single user, there are one of two methods. We could make assumptions about what subsets of variables are part of each sub-utility function, and what those functions are, or the user would need to know and give us this information. We think this is a very difficult task for the user. In our approach, we only do full trace queries and ask for annotations and preference ratings. We think it is more natural to ask the user for their overall rating of a plan, rather than how much each subset of features affected the rating. The neural network then handles the job of learning the preference function over the user-specified features (and any dependencies).The other formalism for specifying preferences are LTL rules (Huth and Ryan 2004) (p.175), which allow the user to specify sequential patterns. Expecting the user to be able to specify LTL rules might be unreasonable. The user would also have to give utilities or preference orderings over the specified LTL rules. One can interpret our interface as extracting a subset of simple LTL rules (through annotations) which are present in a plan trace. The user gives a rating to the trace, as well as what features (LTL rules) were good or bad. Extending the LTL analogy, our encoding of a plan trace can be seen as a vector of the relevant LTL rules. The index corresponding to an LTL rule is set to 1 if the rule is satisfied in the plan trace. However, recall that we also allow cardinal features (counts) in our encoding, and not just binary variables. Our interface and learning framework does not handle the entire gamut of possible LTL rules. We are working on extending the types of sequential preferences supported, while keeping the interface intuitive and expressive. In our approach, we use feature-directed Active Learning complemented with an intuitive and expressive user interface to learn the user's preference function efficiently. The traces obtained during active learning are rated and annotated by the user. These traces are encoded as a vector over the features that the user indicated as relevant to their preferences. The feature vectors are used to train a simple feedforward Neural Network to learn the preference function. We show that the SimpleNN neural network is more accurate and interpretable with fewer, more informative plan traces as compared to the LSTM based SequenceNN model. The latter was trained with a larger dataset of rated plan traces without active learning. Our current experiments use a user preference function over only a few variables. It is important to see how efficiently our framework learns a more complex preference function. Moreover, the current preference function is completely deterministic as it provides consistent annotation and rating to the plan trace. A human, however, might not behave in a consistent manner. We will test with a noisy or probabilistic preference model in future work. The user interface itself can be extended to include more complex annotations. For example, the user can also provide annotations for some features to be added/dropped from the plan. This is especially useful for cardinal feature as the modified feature count represents what is ideal to the user. For example, if the user's preference doesn't increase after visiting more than 2 lakes. Then this can be communicated by removing extra lake features from a plan trace. We have mentioned categorical and cardinal features, but our framework is also intended to support real-valued features. We would need to adapt our active learning process to elicit feedback as to what the minimum, optimum and maximum values of such features are. These would be the minimum essential points to sample for approximating the underlying utility function. Lastly, we would like to simplify the function by which we choose plan traces in successive rounds of active learning. We think that the similarity with traces from previous rounds is unnecessary, and might not appreciably reduce the cognitive load on the user. We think that just diversity and selecting traces that are much more preferred(closer to 1.0) or much less preferred(closer to 0.0) would be sufficient.

Learning preferences over plan traces using active learning.