Question: Represent the scientific article to find a short TLDR

Question: We propose an interactive classification approach for natural language queries. Instead of classifying given the natural language query only, we ask the user for additional information using a sequence of binary and multiple-choice questions. At each turn, we use a policy controller to decide if to present a question or pro-vide the user the final answer, and select the best question to ask by maximizing the system information gain. Our formulation enables bootstrapping the system without any interaction data, instead relying on non-interactive crowdsourcing an-notation tasks. Our evaluation shows the interaction helps the system increase its accuracy and handle ambiguous queries, while our approach effectively balances the number of questions and the final accuracy. Responding to natural language queries through simple, single-step classification has been studied extensively in many applications, including user intent prediction ), and information retrieval . Typical methods rely on a single user input to produce an output, missing an opportunity to interact with the user to reduce ambiguity and improve the final prediction. For example, users may under-specify a request due to incomplete understanding of the domain; or the system may fail to correctly interpret the nuances of the input query. In both cases, a low quality input could be mitigated by further interaction with the user. In this paper we propose a simple but effective interaction paradigm that consists of a sequence of binary and multiple choice questions allowing the system to ask the user for more information. Figure 1 illustrates the types of interaction supported by this method, showcasing the opportunity for clarification while avoiding much of the complexity involved in unrestricted natural language interactions. Following a natural language query from the user, our system then decides between posing another question to obtain more information or finalizing the current prediction. Unlike previous work which assumes access to full interaction data; Rao & Daumé ), we are interested in bootstrapping an interaction system using simple and relatively little annotation effort. This is particularly important in real-world applications, such as in virtual assistants, where the supported classification labels are subject to change and thereby require a lot of re-annotation. We propose an effective approach designed for interaction efficiency and simple system bootstrapping. Our approach adopts a Bayesian decomposition of the posterior distributions over classification labels and user's responses through the interaction process. Due to the decomposition, we can efficiently compute and select the next question that provides the maximal expected information based on the posteriors. To further balance the potential increase in accuracy with the cost of asking additional questions, we train a policy controller to decide whether to ask additional questions or return a final prediction. Our method also enables separately collecting natural language annotations to model the distributions of class labels and user responses. Specifically, we crowdsource initial natural language queries and question-answer pairs for each class label, alleviating the need for Wizard-of-Oz style dialog annotations . Furthermore, we leverage the natural language descriptions of class labels, questions and answers to help estimate their correlation and reduce the need for heavy annotation. Got it! The article below might be helpful: What is the bill length of the bird: shorter, similar, or longer than head? Shorter than head. Is the bird underpart orange? Yes. The identified bird is: Saw a little black bird with black eyes.... Figure 1: Two example use cases of interactive classification system: providing customer the best trouble-shooting suggestion (left) and helping user identify bird species from text interactions (right). The top parts show example classification labels: FAQ documents or bird species, where the ground truth label of each interaction example is shaded. The lower parts show how a user interact with the system typically. The user starts with an initial natural language query. At each step, the system asks a clarification question. The interaction ends when the system returns an output label. We evaluate our method on two public tasks: FAQ suggestion and bird identification using the text and attribute annotations of the Caltech-UCSD Birds dataset . The first task represents a virtual assistant application in a trouble-shooting domain, while the second task provides well-defined multiple-choice question annotations and naturally noisy language inputs. Our experiments show that adding user interactions significantly increases the classification accuracy, when evaluating against both a simulator and a real human. With one clarification question, our system obtains a relative accuracy boost of 40% and 65% for FAQ suggestion and bird identification compared to no-interaction baselines on simulated evaluation. Given at most five turns of interaction, our approach improves accuracy by over 100% on both tasks in both simulated and human evaluation. Notation Our goal is to classify a natural language query as one class label through interactions. To this end, we treat the classification label y, interaction question q as well as the user response r as random variables. We denote one possible value or assignment of the random variable using subscripts, such as y = y i and q = q j. We use superscripts for the observed value of the random variable at a given time step, for example, q t is a question asked at time step t. Whenever it is clear from the context, we simply write y i as the short notation of y = y i. For example, p(r|q j, y i) denotes the conditional distribution of r given y = y i and q = q j, and p(r k |q j, y i) further specifies the corresponding probability when r = r k. An interaction starts with the user providing an initial user query x. At each turn t, the system can choose to select a question q t, to which the user respond with r t. We consider two types of questions in this work: binary and multiple choice questions. The predefined set of possible answers for a question q t is R(q t), where R(q t) is {yes, no} for binary questions, or a predefined set of question-specific values for multiple choice questions. We denote an interaction up to time t as X t = (x, (q 1, r 1),..., (q t, r t)), and the set of possible class labels as Y = {y 1, . . ., y N}. Figure 1 shows exemplary interactions in our two evaluation domains. Model We model the interactive process using a parameterized distribution over class labels that is conditioned on the observed interaction (Section 4.1), a question selection criterion (Section 4.2), and a parameterized policy controller (Section 4.5). At each time step t, we compute the belief of each y i ∈ Y conditioned on X t−1. The trained policy controller decides between two actions: to return the current best possible label or to obtain additional information by asking a question. The model selects the question that maximizes the information gain. After receiving the user response, the model updates the beliefs over the classification labels. Learning We use crowdsourced non-interactive data to bootstrap model learning. The crowdsourced data collection consists of two sub-tasks. First, we obtain a set of user initial queries X i for each label y i. For example, for an FAQ,'How do I sign up for Spring Global Roaming', an annotator can come up with an initial query as'Travel out of country'. Second, we ask annotators to assign text tags to each y i, and convert these tags into a set of question-answer pairs Here q m denotes a templated question and r m denotes the answer. For example, a question' What is your phone operating system?' can pair with one of the following answers:'IOS','Android operating system','Windows operating system' or'Not applicable'. We denote this dataset as. We describe the data collection process in Section 5. We use this data to train our text embedding model (Section 4.3), to create a user simulator (Section 4.4), and to train the policy controller to minimize the number of interaction turns while achieving high classification accuracy (Section 4.5). Evaluation We report classification accuracy of the model, and study the trade-off between accuracy and the number of the turns that the system takes. We run our system against a user simulator and real human users. When performing human evaluation, we additionally collect qualitative ratings of the interactions. Learning from Feedback A lot of recent work have leveraged human feedback to train natural language processing models, including dialogue learning , semantic parsing (; ;) and text classification . These works collect user feedback after the model-predicting stage and treat user feedback as extra offline training data to improve the model. In contrast, our model leverages the user interaction and makes model prediction accordingly in an online fashion. Human feedback has been incorporated in reinforcement learning as well. For instance, learns a reward function from human preferences to provide richer rewards and uses language-illustrated subgoals as indirect interventions instead of conventional expert demonstrations. Modeling Interaction Language-based interaction has recently attracted a lot of attention in areas such as visual question answering (; ; ;), SQL generation , information retrieval and multi-turn textbased question answering (Rao & Daumé ; ;). Many of these works require learning from full-fledged dialogues; Rao & Daumé ) or conducting Wizard-of-Oz dialog annotations . Instead of utilizing unrestricted but expensive conversations, we limit ourselves to a simplified type of interaction consisting of multiple-choice and binary questions. This allows us to reduce the complexity of data annotation while still achieving efficient interaction and high accuracy. Our question selection method is closely related to the prior work of Rao & Daumé;;; ). For instance, refine image search by asking to compare visual qualities against selected reference images, and perform object identification in image by posing binary questions about the object or its location. Both works and our system use an entropy reduction criterion to select the best question. Our work makes use of a Bayesian decomposition of the joint distribution and can be easily extended to other model-driven selection. We also highlight the modeling of natural language to estimate information and probabilities. More recently, (Rao & Daumé) proposes a learning-to-ask model by modeling the expected utility obtained by a question. Our selection method can be considered as a special case when entropy is used as the utility. In contrast to (Rao & Daumé), our work models the entire interaction history instead of a single turn of follow-up questioning. Our model is trained using crowdsourced annotations, while (Rao & Daumé) uses real user-user interaction data. Learning 20Q Game Our task can be viewed as an instance of the popular 20-question game (20Q), which has been applied to the knowledge base of celebrities ). Our work differs in two fold. First, our method models the natural language descriptions of classification targets, quesitons ans answers, instead of treating them as categorical or structural data as in knowledge base. Second, instead of focusing on the 20Q game (on celebrities) itself, we aim to help users accomplish realistic goals with minimal interaction effort. We maintain a probability distribution p(y | X t) over the set of labels Y. At each interaction step, we first update this belief, decide if to ask the question or return the classification output using a policy controller and select a question to ask using information gain if needed. We decompose the conditional probability p(y = y i | X t) using Bayes rule: We make two simplifying assumptions for Eq.. First, we assume the user response only depends on the provided question q t and the underlying target label y i, and is independent of past interactions. The assumption simplifies p(r Second, we deterministically select the next question q t given the interaction history X t−1 (described in Section 4.2). As a , p(q = q t | y i, X t−1) = 1 and otherwise would be zero if q = q t. These two assumptions allow us to rewrite the decomposition as: That is, predicting the classification label given the observed interaction X t can be reduced to modeling p(y i | x) and p(r k | q j, y i), i.e. the label probability given the initial query only and the probability of user response conditioned on the chosen question and class label. This factorization enables us to leverage separate annotations to learn these two components directly, alleviating the need for collecting costly full user interactions. The system selects the question q t to ask at turn t to maximize the efficiency of the interaction. We use a maximum information gain criterion. Given X t−1, we compute the information gain on classification label y as the decrease on entropy by observing possible answers to question q: where H(· | ·) denotes the conditional entropy. Intuitively, the information gain measures the amount of information obtained about the variable y by observing the value of another variable q. Because the first entropy term H(y | X t−1) is a constant regardless of the choice of q, the selection of q t is equivalent to Here we use the independent assumption stated in Section 4.1 to calculate p(r k | X t−1, q j). Both p(r k | X t−1, q j) and p(y i | X t−1, q j, r k) can be iteratively updated utilizing p(y i | x) and p(r k | q j, y i) as the interaction progresses (Eq. 2), ing in efficient computation of information gain. We model p(y i | x) and p(r k | q j, y i) by encoding the natural language descriptions of questions, answers and classification labels 1. That is, we do not simply treat the labels, questions and answers as categorical variables in our model. Instead, we leverage their natural language aspects to better estimate their correlation, reduce the need for heavy annotation and improve our model in lowresource (and zero-shot) scenarios. Specifically, we use a shared neural encoder enc(·) to encode all texts. Both probability distributions are computed using the dot-product score, i.e. S(u, v) = enc(u) enc(v) where u and v are two pieces of text. The probability of predicting the label y i given an initial query x is:. The probability of an answer r k given a question q j and label y i is a linear combination of the observed empirical probabilityp(r k | q j, y i) and a parameterized estimationp(r k | q j, y i): where λ ∈ is a hyper-parameter. We use the question-answer annotations A i for each label y i to estimatep(r k | q j, y i) using the empirical count. For example, in the FAQ suggestion task, we collect multiple user responses for each question and class label, and average across annotator to estimatep (Section 5). The second termp(r k | q j, y i) is estimated using texts:, where q j #r k is a concatenation of the question q j and the answer r k 2 and w, b ∈ R are scalar parameters. Because we do not collect complete annotations to cover every label-question pair,p provides a smoothing of the partially observed counts using the learned encoding S(·). We estimate the enc(·) with parameters ψ by pre-training using the dataset,, as described earlier. We use this data to create a set of text pairs (u, v) to train the scoring function S(·). For each label y i, we create pairs (x, y i) with all its initial queries x ∈ X i. We also create (q m #r m, y i) for each question-answer pair (q m, r m) annotated with the label y i. We perform gradient descent to minimize the cross-entropy loss: The second term requires summation over all v, which are all the labels in Y. We approximate this sum using negative sampling that replaces the full set Y with a sampled subset in each training batch. The parameters ψ, w and b are fine-tuned using reinforcement learning during training of the policy controller (Section 4.5). The user simulator provides initial queries to the system, responds to the system initiated clarification questions and judges if the returned label is correct. The simulator is based on held-out dataset of tuples of a goal y i, a set of initial queries X i, and a set of question answer pairs m=1. We estimate the simulator response distribution p (r k | q j, y i) using smoothed empirical counts from the data. While this data is identical in structure to our training data, we keep it separated from the data used to estimate S(·), p(y i | x) and p(r k | q j, y i) (Section 4.3). At the beginning of an interaction, the simulator selects a target labelŷ, and samples a query x from the associated query set to start the interaction. Given a system clarification question q t at turn t, the simulator responds with an answer r t ∈ R(q t) by sampling from a belief distribution p (r | q t,ŷ). Sampling provides natural noise to the interaction, and our model has no knowledge of p. The interaction ends when the system returns a target. This setup is flexible in that the user simulator can be easily replaced or extended by a real human, and the system can be further trained with the human-in-the-loop setup. The policy controller decides at each turn t to either select another question to query the user or to conclude the interaction. This provides a trade-off between exploration by asking questions and exploitation by returning the most probable classification label. The policy controller f (·, ·; θ) is a feed-forward network parameterized by θ that takes the top-k probability values and current turn t as input state. It generates two possible actions, STOP or ASK. When the action is ASK, a question is selected to maximize the information gain, and when the action is STOP, the label y i with highest probability is returned using arg max yi∈Y p(y i | X t−1). We tune the policy controller using the user simulator (Section 4.4). Algorithm 1 describes the training process. During learning, we use a reward function that provides a positive reward for predicting the correct target at the end of the interaction, a negative reward for predicting the wrong target, and a small negative reward for every question asked. We learn the policy controller f (·, ·; θ), and fine-tune p(r k | q j, y i) by back-propagating through the policy gradient. We keep the enc(·) parameters fixed during this process. We design a crowdsourcing process to collect data for the FAQ task using Amazon Mechanical Turk 3. For the Birds domain, we re-purpose an existing dataset. We collect initial queries and tags for each FAQ document. We ask workers to consider the scenario of searching for an FAQ supporting document using an interactive system. Given a target FAQ, we ask for an initial query that they would provide to such a system. The set of initial queries that is collected for each document y i is X i. We encourage workers to provide incomplete information and avoid writing a simple paraphrase of the FAQ. This in more realistic and diverse utterances because users have limited knowledge of the system and the domain. We collect natural language tag annotations for the FAQ documents. First, we use domain experts to define the set of possible free-form tags. The tags are not restricted to a pre-defined ontology and can be a phrase or a single word, which describes the topic of the document. We heuristically remove duplicate tags to finalize the set. Next, experts heuristically combine some tags to categorical tags, while leave all the rest tags as binary tags. For example, tags'IOS','Android operating system' and'Windows operating system' are combined to form a categorical tag'phone operating system'. We then use a small set of deterministic, heuristically-designed templates to convert tags into questions. For example, the tag'international roaming' is converted into a binary question' Is it about international roaming?'; the categorical tag'phone operating system' is converted into a multi-choice question' What is your phone operating system?'. Finally, we use non-experts to collect user responses to the questions of the FAQ. For binary questions, we ask workers to associate the tags to the FAQ target if they would respond'yes' to the question. We show the workers a list of ten tags for a given target as well as'none of the above' option. Annotating all target-tag combinations is excessively expensive and most pairings are negative. We rank the tags based on the relevance against the target using S(·) and show only top-50 to the workers. For multi-choice tags, we show the workers a list of possible answers to a tag-generated question for a given FAQ. The workers need to choose one answer that they think best applies. They also have the option of choosing'not applicable'. We provide more data collection statistics in Appendix A.1. The workers do not engage in a multi-round interactive process. This allows for cheap and scalable collection. Task I: FAQ Suggestion We use the FAQ dataset from . The dataset contains 517 troubleshooting documents by crawling Sprint's technical website. In addition, we collect 3, 831 initial queries and 118, 640 tag annotations using the setup described in Section 5. We split the data into 310/103/104 documents as training, development, and test sets. Only the queries and tag annotations of the 310 training documents are used for pre-training and learning the policy controller. We use the queries and tag annotations of the development and test documents for evaluation only. The classification targets contain all 517 documents during evaluation 4. Task II: Bird Identification Our second set of experiments uses the Caltech-UCSD Birds (CUB-200) dataset . The dataset contains 11, 788 bird images for 200 different bird species. Each bird image is annotated with a subset of 27 visual attributes and 312 attribute values pertaining to the color or shape of a particular part of the bird. We take attributes with value count less than 5 as categorical tags, leaving us 8 categorical questions in total. The remaining 279 attributes are treated as binary tags and converted to binary questions. In addition, each image is annotated with 10 image captions describing the bird in the image . We use the image captions as initial user queries and bird species as labels. Since each image often contains only partial information about the bird species, the data is naturally noisy and provides challenging user interactions. We do not use the images from the dataset for model training. The images are only provided for grounding during human evaluation. Baselines We compare our full model against the following baseline methods: • No Interact: the best classification label is predicted using only the initial query. We consider two possible implementations: BM25, a common keyword-based scoring model for retrieval methods , and a neural model described in Section 4.3. • Random Interact: at each turn, a random question is chosen and presented to the user. After T turns, the classification label is chosen according to the belief p(y | X T). • Static Interact: questions are picked without conditioning on the initial user query using maximum information criterion. This is equivalent to using a static decision tree to pick the question, leading to the same first question, similar to . • Variants of Ours: we consider several variants of our full model. First, we replace the policy controller with two termination strategies: one which ends interaction when max p(y | X t) passes a threshold, and another one which ends interaction after the designated number of turns. Second, we disable the parameterized estimatorp(r k | q j, y i) by setting λ to 1. Evaluation We evaluate our model by running against both the user simulator and a real human. Given the user simulator, we evaluate the classification performance of our model and baselines using Accuracy@k, which is the percentage of time the correct target appears among the top-k predictions of the model. During human evaluation, we ask annotators to interact with our proposed or baseline models through a web-based interactive interface. Each interaction session starts with a user scenario 5 presented to an annotator (e.g a bird image or a device-troubleshooting scenario described in text). The annotator inputs an initial query accordingly and then answers follow-up questions selected by the system. Once the system returns its prediction, the annotator is asked to provide a few ratings of the interaction, such as rationality -do you feel that you were understood by the system?. We present more details of the human evaluation in Appendix A.3. We use a fast recurrent neural network to encode texts . The policy controller receives three different rewards: a positive reward for returning the correct target (r p = 20), a negative reward for providing the wrong target (r n = −10) and a turn penalty for each question asked (r a = −1, . . ., −5). We report the averaged over 3 independent runs for each model variant and baseline. More details about the model implementation and training procedure can be found in Appendix A.2. Bird Identification 38 ± 0.5% 61 ± 0.3% 23 ± 0.1% 41 ± 0.2% Random Interact 39 ± 0.3% 62 ± 0.4% 25 ± 0.1% 44 ± 0.1% Static Interact 46 ± 0.5% 66 ± 0.6% 29 ± 0.2% 50 ± 0.3% Ours 79 ± 0.7% 86 ± 0.8 % 49 ± 0.3 % 69 ± 0.5 % w/ threshold 72 ± 0.6% 82 ± 0.7% 41 ± 0.3% 59 ± 0.4% w/ fixed turn 71 ± 1.0% 81 ± 0.9% 39 ± 0.2% 56 ± 0.4% w/ λ = 1 66 ± 0.8% 71 ± 1.0% 40 ± 0.1% 56 ± 0.2% Table 1: Performance of our system against various baselines, which are evaluated using Accuracy@{1, 3}. For all interacting baselines, 5 clarification questions are used. Best performances are in bold. We report the averaged as well as the standard deviations from 3 independent runs for each model variant and baseline. Table 2: Human evaluation . Count is the total number of interaction examples. The system is evaluated with Accuracy@1 and the rationality score ranging from −2 (strongly disagree) to 2 (strongly agree). Table 1 shows the performance of our model against the baselines on both tasks while evaluating against user simulator. The No Interact (Neural) baseline achieves an Accuracy@1 of 38% for FAQ Suggestion and 23% for Bird Identification. The No Interact (BM25) baseline performs worst. The Random Interact baseline and the Static Interact baseline barely improve the performance, illustrating the challenge of building an effective interactive model. In contrast, our model and its variants obtain substantial gain in accuracy given a few turns of interaction. Our full model achieves an Accuracy@1 of 79% for FAQ Suggestion and 49% for Bird Identification using less than 5 turns, outperforming the No Interact (Neural) baseline by an absolute number of 41% and 26%. The two baselines with alternative termination strategies underperform the full model, indicating the effectiveness of the policy controller trained with reinforcement learning. The model variant with λ = 1, which has fewer probability components leveraging natural language than our full model, achieves worse Accuracy@1. This , together with the fact that our model outperforms the Static Interact baseline, confirms the importance of modeling natural language for efficient interaction. Figure 2 shows the trade-off between classification accuracy and the number of the turns the system takes. The number of interactions changes as we vary the model termination strategy, which includes varying turn penalty r a, the prediction threshold, and the predefined number of turns T. Our model with the policy controller or the threshold strategy does not explicitly control the number of turns, so we report the average number of turns across multiple runs for these two models. We achieve a relative accuracy boost of 40% for FAQ Suggestion and 65% for Bird Identification over no-interaction baselines with only one clarification question. This highlights the value of leveraging human feedback to improve model accuracy in classification tasks. Our approach outperforms baselines ranging across all numbers of interactions. Table 2 shows the human evaluation of our full model and two baselines on the FAQ Suggestion and Bird Identification tasks. The model achieves Accuracy@1 of 30% and 20% for FAQ and Bird tasks respectively, when there is no interaction. Each of the model variants uses 3 interaction turns on average, and all three models improve the classification after the interaction. Our full model achieves the best performance: an Accuracy@1 of 59% for FAQ Suggestion and 45% for Bird Identification. Qualitatively, the users rate our full model to be more rational. The human evaluation demonstrates that our model handles real user interaction more effectively despite being trained with only non-interactive data. Appendix A.3 includes additional details for human evaluation and exemple interactions. Figure 3 shows the learning curves of our model with the policy controller trained with different turn penalty r a ∈ {−0.5, −1, −3}. We observe interesting exploration behavior during the first 1, 000 training episodes in the middle and the right plots. The models achieve relatively stable accuracy after the early exploration stage. As expected, the three runs end up using different numbers of expected turns due to the choice of different r a values. We propose an approach for interactive classification, where users can provide under-specified natural language queries and the system can inquire missing information through a sequence of simple binary or multi-choice questions. Our method uses information theory to select the best question at every turn, and a lightweight policy to efficiently control the interaction. We show how we can bootstrap the system without any interaction data. We demonstrate the effectiveness of our approach on two tasks with different characteristics. Our show that our approach outperforms multiple baselines by a large margin. In addition, we provide a new annotated dataset for future work on bootstrapping interactive classification systems. A.1 DATA COLLECTION Query collection qualification One main challenge for the collection process lies within familiarizing the workers with the set of target documents. To make sure we get good quality annotation, we set up a two-step qualification task. The first one is to write paraphrase with complete information. After that, we reduce the number of workers down to 50. These workers then generate 19, 728 paraphrase queries. During the process, the workers familiarize themselves with the set of documents. We then post the second task (two rounds), where the workers try to provide initial queries with possibly insufficient information. We select 25 workers after the second qualification task and collect 3, 831 initial queries for the second task. Attribute Collection Qualification To ensure the quality of target-tag annotation, we use the pretrained model to rank-order the tags and pick out the highest ranked tags (as positives) and the lowest ranked tags (as negatives) for each target. The worker sees in total ten tags without knowing which ones are the negatives. To pass the qualifier, the workers need to complete annotation on three targets without selecting any of the negative tags. To make the annotation efficient, we rank the tag-document relevance using the model trained on the previously collected query data. We then take the top 50 possible tags for each document and split them into five non-overlapping lists (i.e. ten tags for each list). Each of the list is assigned to four separate workers to annotate. In the FAQ task, we observe that showing only the top-50 tags out of 813 is sufficient. Table A.1: Target-tag annotation statistics. We show five sets of tags to the annotators. The higher ranked ones are more likely to be related to the given target. The row mean # tags is the mean number of tags that are annotated to a target, N.A. is the percentage of the tasks are annotated as "none of the above", and mean κ is the mean pairwise Cohen's κ score. Learning Components Here we describe the detailed implementation of the text encoder and the policy controller network. We use a single-layer bidirectional Simple Recurrent Unit (SRU) as the encoder for the FAQ suggestion task and two layer bidirectional SRU for bird identification task. The encoder uses pre-trained fastText word embedding of size 300 (fixed during training), hidden size 150, batch size 200, and dropout rate 0.1. The policy controller is a two layer feed-forward network with hidden layer size of 32 and ReLU activation function. We use Noam learning rate scheduler with initial learning rate 1e − 3, warm-up step 4, 000 and Noam scaling factor 2.0. The policy controller is a 2 layer feed-forward network with a hidden layer of 32 dimensions and ReLU activation. The network takes the current step and the top-k values of belief probabilities as input. We choose k = 20 and allow a maximum of 10 interaction turns. We use initial queries as well as paraphrase queries to train the encoder, which has around 16K target-query examples. The breakdown analysis is shown in Table A.2. To see the effectiveness of the tag in addition to initial query, we generate pseudo-queries by combining existing queries with sampled subset of tags from the targets. This augmentation strategy is shown to be useful to improve the classification performance. On the other hand, when we use the set of tags instead of initial query as text input for a specific target label, the classification performance improves, indicating the designed tags can capture the target label well. Finally, when we concatenate user initial query and tags and use that as text input to the classifier, we achieve Accuracy@1 of 76%. In our full model, we achieve 79% with only querying about 5 tags, indicating effectiveness of our modelling. Each interaction session starts with presenting the annotator an user scenario (e.g a bird image or an issue with your phone). The annotator inputs an initial query accordingly and then answers follow-up questions selected by the system. We evaluate prediction accuracy, system rationality, and the number of counts by letting the system interact with human judges. We design user scenario for each target to present to the worker. At the end of each interaction, the predicted FAQ and the ground truth will be presented to the user as shown in the top right panel in Figure A.2. The user needs to answer the following questions: "How natural is the interaction?" and "Do you feel understood by the system during the interactions?" on the scale of −2 (strongly disagree) to 2 (strongly agree), which we record as naturalness and rationality in Table A. 3. Our full model performs best on Accuracy@1, naturalness, and rationality. We show human evaluation examples in Table A.4. The interface for bird identification task is similar to the FAQ suggestion task. Instead of presenting a scenario, we show a bird image to the user. The user needs to describe the bird to find out its category, which is analogous to writing an initial query. We allow the user to reply'not visible' if part of the bird is hidden or occluded. With such reply, the system stops asking binary questions from the same label group. For example, if user replied'not visible' to a the question'does the bird has black tail?', then question'does the bird has yellow tail?','does the bird has red tail?' etc. won't be asked again. At the end of the interaction, the predicted and ground-truth bird images along with their categories are presented to the user as shown in the bottom right panel in Figure A.2. Again, the user needs to fill out a similar questionnaire as in FAQ suggestion task. The bird identification task is very challenging due to its fine-grained categories, where many bird images look almost identical while belonging to different classes. Our full system improves Accuracy@1 from 20% to 45% against non-interactive baselines after less than 3 turns of interaction. For Bird Identification, the annotators reported that the predicted image is almost identical to the true image sometimes. To better understand the task and the model behavior, we show the confusion matrix of the final model prediction after interaction in Figure A Table A.3: Human evaluation on FAQ Suggestion and Bird Identification on our proposed model and several baslines. The three FAQ systems ask 2.8, 3 and 3 turns of questions, respectively. The three Bird systems ask 3.3, 4 and 4 turns of questions. The system is evaluated with both on performance and user experience. Performance include the initial and final Accuracy@1. The user experience score include both naturalness and rationality for both task. We desire high value in the diagonal part and low value elsewhere.

Represent the short summary for finding the corresponding scientific article (long)

We propose an interactive approach for classifying natural language queries by asking users for additional information using information gain and a reinforcement learning policy controller.

Introduce an approach to allow agents to learn  PPDDL action models incrementally over multiple planning problems under the framework of reinforcement learning.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Convolutional neural networks (CNNs) have achieved state of the art performance on recognizing and representing audio, images, videos and 3D volumes; that is, domains where the input can be characterized by a regular graph structure. However, generalizing CNNs to irregular domains like 3D meshes is challenging. Additionally, training data for 3D meshes is often limited. In this work, we generalize convolutional autoencoders to mesh surfaces. We perform spectral decomposition of meshes and apply convolutions directly in frequency space. In addition, we use max pooling and introduce upsampling within the network to represent meshes in a low dimensional space. We construct a complex dataset of 20,466 high resolution meshes with extreme facial expressions and encode it using our Convolutional Mesh Autoencoder. Despite limited training data, our method outperforms state-of-the-art PCA models of faces with 50% lower error, while using 75% fewer parameters. Convolutional neural networks BID27 have achieved state of the art performance in a large number of problems in computer vision BID26 BID22, natural language processing BID32 and speech processing BID20. In recent years, CNNs have also emerged as rich models for generating both images BID18 and audio. These successes may be attributed to the multi-scale hierarchical structure of CNNs that allows them to learn translational-invariant localized features. Since the learned filters are shared across the global domain, the number of filter parameters is independent of the domain size. We refer the reader to BID19 for a comprehensive overview of deep learning methods and the recent developments in the field. Despite the recent success, CNNs have mostly been successful in Euclidean domains with gridbased structured data. In particular, most applications of CNNs deal with regular data structures such as images, videos, text and audio, while the generalization of CNNs to irregular structures like graphs and meshes is not trivial. Extending CNNs to graph structures and meshes has only recently drawn significant attention BID8 BID14. Following the work of BID14 on generalizing the CNNs on graphs using fast Chebyshev filters, we introduce a convolutional mesh autoencoder architecture for realistically representing high-dimensional meshes of 3D human faces and heads. The human face is highly variable in shape as it is affected by many factors such as age, gender, ethnicity etc. The face also deforms significantly with expressions. The existing state of the art 3D face representations mostly use linear transformations BID39 BID29 BID40 or higher-order tensor generalizations BID43 BID9. While these linear models achieve state of the art in terms of realistic appearance and Euclidean reconstruction error, we show that CNNs can perform much better at capturing highly non-linear extreme facial expressions with many fewer model parameters. One challenge of training CNNs on 3D facial data is the limited size of current datasets. Here we demonstrate that, since these networks have fewer parameters than traditional linear models, they can be effectively learned with limited data. This reduction in parameters is attributed to the locally invariant convolutional filters that can be shared on the surface of the mesh. Recent work has exploited thousands of 3D scans and 4D scan sequences for learning detailed models of 3D faces BID13 BID46 BID37 BID11. The availability of this data enables us to a learn rich non-linear representation of 3D face meshes that can not be captured easily by existing linear models. In summary, our work introduces a convolutional mesh autoencoder suitable for 3D mesh processing. Our main contributions are:• We introduce a mesh convolutional autoencoder consisting of mesh downsampling and mesh upsampling layers with fast localized convolutional filters defined on the mesh surface.• We use the mesh autoencoder to accurately represent 3D faces in a low-dimensional latent space performing 50% better than a PCA model that is used in state of the art methods BID39 for face representation.• Our autoencoder uses up to 75% fewer parameters than linear PCA models, while being more accurate on the reconstruction error.• We provide 20,466 frames of highly detailed and complex 3D meshes from 12 different subjects for a range of extreme facial expressions along with our code for research purposes. Our data and code is located at http://withheld.for.review.This work takes a step towards the application of CNNs to problems in graphics involving 3D meshes. Key aspects of such problems are the limited availability of training data and the need for realism. Our work addresses these issues and provides a new tool for 3D mesh modeling. Mesh Convolutional Networks. give a comprehensive overview of generalizations of CNNs on non-Euclidean domains, including meshes and graphs. defined the first mesh convolutions by locally parameterizing the surface around each point using geodesic polar coordinates, and defining convolutions on the ing angular bins. In a follow-up work, BID4 parametrized local intrinsic patches around each point using anisotropic heat kernels. BID33 introduced d-dimensional pseudo-coordinates that defined a local system around each point with weight functions. This method resembled the intrinsic mesh convolution of and BID4 for specific choices of the weight functions. In contrast, Monti et al. used Gaussian kernels with trainable mean vector and covariance matrix as weight functions. In other work, BID42 presented dynamic filtering on graphs where filter weights depend on the input data. The work however did not focus on reducing the dimensionality of graphs or meshes. BID45 also presented a spectral CNN for labeling nodes which did not involve any dimensionality reduction of the meshes. BID38 and BID30 embedded mesh surfaces into planar images to apply conventional CNNs. Sinha et al. used a robust spherical parametrization to project the surface onto an octahedron, which is then cut and unfolded to form a squared image. BID30 introduced a conformal mapping from the mesh surface into a flat torus. Although, the above methods presented generalizations of convolutions on meshes, they do not use a structure to reduce the meshes to a low dimensional space. The proposed mesh autoencoder efficiently handles these problems by combining the mesh convolutions with efficient meshdownsampling and mesh-upsampling operators. Graph Convolutional Networks. BID8 proposed the first generalization of CNNs on graphs by exploiting the connection of the graph Laplacian and the Fourier basis (see Section 3 for more details). This lead to spectral filters that generalize graph convolutions. extended this using a windowed Fourier transform to localize in frequency space. BID23 built upon the work of Bruna et al. by adding a procedure to estimate the structure of the graph. To reduce the computational complexity of the spectral graph convolutions, BID14 approximated the spectral filters by truncated Chebyshev poynomials which avoids explicitly computing the Laplacian eigenvectors, and introduced an efficient pooling operator for graphs. BID25 simplified this using only first-order Chebyshev polynomials. However, these graph CNNs are not directly applied to 3D meshes. Our mesh autoencoder is most similar to BID14 with truncated Chebyshev polynomials along with the efficient graph pooling. In addition, we define mesh upsampling layer to obtain a complete mesh autoencoder structure and use our model for representation of highly complex 3D faces obtained state of the art in realistic modeling of 3D faces. Learning Face Representations. BID2 introduced the first generic representation for 3D faces based on principal component analysis (PCA) to describe facial shape and texture variations. We also refer the reader to BID10 for a comprehensive overview of 3D face representations. Representing facial expressions with linear spaces has given state-of-the-art till date. The linear expression basis vectors are either computed using PCA (e.g. BID1 BID6 BID29 BID39 BID44, or are manually defined using linear blendshapes (e.g. BID40 BID28 BID5 . Multilinear models BID43, i.e. higher-order generalizations of PCA are also used to model facial identity and expression variations. In such methods, the model parameters globally influence the shape, i.e. each parameter affects all the vertices of the face mesh. To capture localized facial details, BID34 and BID15 used sparse linear models. BID9 used a hierarchical multiscale approach by computing localized multilinear models on wavelet coefficients. BID9 also used a hierarchical multi-scale representation, but their method does not use shared parameters across the entire domain. BID24 use a volumetric face representation in their CNN-based framework. In contrast to existing face representation methods, our mesh autoencoder uses convolutional layers to represent faces with significantly fewer parameters. Since, it is defined completely on the mesh space, we do not have memory constraints which effect volumetric convolutional methods for representing 3D models. We define a face mesh as a set of vertices and edges F = (V, A), with |V| = n vertices that lie in 3D Euclidean space, V ∈ R n×3. The sparse adjacency matrix A ∈ {0, 1} n×n represents the edge connections, where A ij = 1 denotes an edge connecting vertices i and j, and A ij = 0 otherwise. The non-normalized graph Laplacian is defined as L = D − A BID12, with the diagonal matrix D that represents the degree of each vertex in V as D ii = j A ij. n×n is a diagonal matrix with the associated real, nonnegative eigenvalues. The graph Fourier transform BID12 of the mesh vertices x ∈ R n×3 is then defined as x ω = U T x, and the inverse Fourier transform as x = U x ω, respectively. Fast spectral convolutions. The convolution operator * can be defined in Fourier space as a Hadamard product, DISPLAYFORM0. This is computationally expensive with large number of vertices. The problem is addressed by formulating mesh filtering with a kernel g θ using a recursive Chebyshev polynomial BID21 BID14. The filter g θ is parametrized as a Chebyshev polynomial of order K given by DISPLAYFORM1 whereL = 2L/λ max − I n is the scaled Laplacian, the parameter θ ∈ R K is a vector of Chebyshev coefficients, and T k ∈ R n×n is the Chebyshev polynomial of order k that can be computed recursively as T k (x) = 2xT k−1 (x) − T k−2 (x) with T 0 = 1 and T 1 = x. The spectral convolution can then be defined as BID14 Table 2: Decoder architecture that computes the j th feature of y ∈ R n×Fout. The input x ∈ R n×Fin has F in features. The input face mesh has F in = 3 features corresponding to its 3D vertex positions. Each convolutional layer has F in × F out vectors of Chebyshev coefficients θ i,j ∈ R K as trainable parameters. DISPLAYFORM2 Mesh Sampling The mesh sampling operators define the downscaling and upscaling of the mesh features in a neural net. We perform the in-network downsampling of a mesh with m vertices using transform matrices Q d ∈ {0, 1} n×m, and upsampling using Q u ∈ R m×n where m > n. The downsampling is obtained by contracting vertex pairs iteratively that maintains surface error approximations using quadric matrices BID16. The vertices after downsampling are a subset of the original mesh vertices DISPLAYFORM3 Since a loss-less downsampling and upsampling is not feasable for general surfaces, the upsampling matrix is built during downsampling. Vertices kept during downsampling are kept during upsampling DISPLAYFORM4 Vertices v q ∈ V discarded during downsampling where Q d (p, q) = 0 ∀p, are mapped into the downsampled mesh surface. This is done by projecting v q into the closest triangle (i, j, k) in the downsampled mesh, denoted by v p, and computing the Barycentric coordinates as DISPLAYFORM5 The weights are then updated in Q u as Q u (q, i) = w i, Q u (q, j) = w j, and Q u (q, k) = w k, and Q u (q, l) = 0 otherwise. Figure 1: The effect of downsampling (red arrows) and upsampling (green arrows) on 3D face meshes. The reconstructed face after upsampling maintains the overall structure but most of the finer details are lost. Now that we have defined the basic operations needed for our neural network in Section 3, we can construct the architecture of the convolutional mesh autoencoder. The structure of the encoder is shown in Table 1. The encoder consists of 4 Chebyshev convolutional filters with K = 6 Chebyshev polynomials. Each of the convolutions is followed by a biased ReLU BID17. The downsampling layers are interleaved between convolution layers. Each of the downsampling layers reduce the number of mesh vertices by 4 times. The encoder transforms the face mesh from R n×3 to an 8 dimensional latent vector using a fully connected layer at the end. The structure of the decoder is shown in Table 2. The decoder similarly consists of a fully connected layer that transforms the latent vector from R 8 to R 32×32 that can be further upsampled to reconstruct the mesh. Following the decoder's fully connected layer, 4 convolutional layers with interleaved upsampling layers generated a 3D mesh in R 8192×3. Each of the convolutions is followed by a biased ReLU similar to the encoder network. Each upsampling layer increases the numbers of vertices by 4x. The FIG0 shows the complete structure of our mesh autoencoder. Training. We train our autoencoder for 300 epochs with a learning rate of 8e-3 with a learning rate decay of 0.99 every epoch. We use stochastic gradient descent with a momentum of 0.9 to optimize the L1 loss between predicted mesh vertices and the ground truth samples. We use a regularization on the weights of the network using weight decay of 5e-4. The convolutions use Chebyshev filtering with K = 6. Facial Expression Dataset. Our dataset consists of 12 classes of extreme expressions from 12 different subjects. These expressions are highly complex and uncorrelated with each other. The expressions in our dataset are -bareteeth, cheeks in, eyebrow, high smile, lips back, lips up, mouth down, mouth extreme, mouth middle, mouth side and mouth up. The number of frames of each sequence is shown in TAB2.The data is captured at 60fps with a multi-camera active stereo system (3dMD LLC, Atlanta) with six stereo camera pairs, five speckle projectors, and six color cameras. Our dataset contains 20,466 3D Meshes, each with about 120,000 vertices. The data is pre-processed using a sequential mesh registration method BID29 to reduce the data dimensionality to 5023 vertices. We preprocess the data by adding fake vertices to increase the number of vertices to 8192. This enables pooling and upsampling of the mesh across the layers with a constant factor. Implementation details We use Tensorflow BID0 for our network implementation. We use Scikit-learn BID36 for computing PCA coefficients. Training each network takes about 8 hours on a single Nvidia Tesla P100 GPU. Each of the models is trained for 300 epochs with a batch size of 16. We evaluate the performance of our model based on its ability to interpolate the training data and extrapolate outside its space. We compare the performance of our model with a PCA model. We consistently use an 8-dimensional latent space to encode the face mesh using both the PCA model and the Mesh Autoencoder. Thus, the encoded latent vectors lie in R 8. Meanwhile, the number of parameters in our model is much smaller than PCA model (Table 4).In order to evaluate the interpolation capability of the autoencoder, we split the dataset in training and test samples in the ratio of 1:9. The test samples are obtained by picking consecutive frames of length 10 uniformly at random across the sequences. We train our autoencoder for 300 epochs and evaluate it on the test set. We use mean Euclidean distance for comparison with the PCA method. The mean Euclidean distance of N test mesh samples with n vertices each is given by DISPLAYFORM0 where x ij,x ij ∈ R 3 are j-th vertex predictions and ground truths respectively corresponding to i-th sample. Table 4 shows the mean Euclidean distance along with standard deviation in the form [µ ± σ]. The median error is also shown in the table. We show a performance improvement, as high as 50% over PCA models for capturing these highly non linear facial expressions. At the same time, the number of parameters in the CNN is about 75% fewer than the PCA model as shown in Table 4. Visual inspection of our qualitative in Figure 3 show that our reconstructions are more realistic and are effective in capturing extreme facial expressions. We also show the histogram of cumulative errors in FIG1. We observe that Mesh Autoencoder has about 76.9% of the vertices within an Euclidean error of 2 mm, as compared to 51.7% for the PCA model. To measure generalization of our model, we compare the performance of our model with a PCA model and FLAME BID29. For comparison, we train the expression and jaw model of FLAME with our dataset. The FLAME reconstructions are obtained with with latent vector size of 16 with 8 components each for encoding identity and expression. The latent vectors encoded using PCA model and Mesh autoencoder have a size of 8.We evaluate the generalization capability of the Mesh Autoencoder by attempting to reconstruct the expressions that are completely unseen by our model. We split our dataset by completely excluding one expression set from all the subjects of the dataset. We test our Mesh Autoencoder on the excluded expression as the test set. We compare the performance of our model with PCA and FLAME using the same mean Euclidean distance. We perform 12 cross validation experiments, one for each expression as shown in Table 5. For each experiment, we run our training procedure ten times initializing the weights at random. We pick the best performing network for comparison. We compare the using mean Euclidean distance and median error metric in Table 5. Our method performs better than PCA model and FLAME BID29 on all expression sequences. We show the qualitative in FIG3. Our model performs much better on these extreme expressions. We show the cumulative euclidean error histogram in FIG1. For a 2 mm accuracy, Mesh Autoencoder captures 84.9% of the vertices while the PCA model captures 73.6% of it. The FLAME model BID29 uses several PCA-models to represent expression, jaw motion, face identity etc. We evaluate the performance of mesh autoencoders by replacing the expression model of FLAME by our autoencoder. We compare the reconstruction errors with the original FLAME model. We run our experiment by varying the size of the latent vector for encoding. We show the comparisons in Table 6. While our convolutional Mesh Autoencoder leads to a representation that generalizes better for unseen 3D faces than PCA with much fewer parameters, our model has several limitations. Our network is restricted to learning face representation for a fixed topology, i.e., all our data samples needs to have the same adjacency matrix, A. The mesh sampling layers are also based on this fixed adjacency matrix A, which defines only the edge connections. The adjacency matrix does not take in to account the vertex positions thus affecting the performance of our sampling operations. In future, we would like to incorporate this information into our learning framework. Mesh Autoencoder PCA FLAME BID29 Table 5: Quantitative evaluation of Extrapolation experiment. The training set consists of the rest of the expressions. Mean error is of the form [µ ± σ] with mean Euclidean distance µ and standard deviation σ. The median error and number of frames in each expression sequnece is also shown. All errors are in millimeters (mm).The amount of data for high resolution faces is very limited. We believe that generating more of such data with high variability between faces would improve the performance of Mesh Autoencoders for 3D face representations. The data scarcity also limits our ability to learn models that can be trained for superior performance at higher dimensional latent space. The data scarcity also produces noise in some reconstructions. We have introduced a generalization of convolutional autoencoders to mesh surfaces with mesh downsampling and upsampling layers combined with fast localized convolutional filters in spectral space. The locally invariant filters that are shared across the surface of the mesh significantly reduce the number of filter parameters in the network. While the autoencoder is applicable to any class of mesh objects, we evaluated its quality on a dataset of realistic extreme facial expressions. Table 6: Comparison of FLAME and FLAME++. FLAME++ is obtained by replacing expression model of FLAME with our mesh autoencoder. All errors are in millimeters (mm).convolutional filters capture a lot of surface details that are generally missed in linear models like PCA while using 75% fewer parameters. Our Mesh Autoencoder outperforms the linear PCA model by 50% on interpolation experiments and generalizes better on completely unseen facial expressions. Face models are used in a large number of applications in computer animations, visual avatars and interactions. In recent years, a lot of focus has been given to capturing highly detailed static and dynamic facial expressions. This work introduces a direction in modeling these high dimensional face meshes that can be useful in a range of computer graphics applications.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Convolutional autoencoders generalized to mesh surfaces for encoding and reconstructing extreme 3D facial expressions.

We propose a SVD based method to explore the local dimension of activation manifold in deep neural networks.


Question: Represent the input

Question: Computing distances between examples is at the core of many learning algorithms for time series. Consequently, a great deal of work has gone into designing effective time series distance measures. We present Jiffy, a simple and scalable distance metric for multivariate time series. Our approach is to reframe the task as a representation learning problem---rather than design an elaborate distance function, we use a CNN to learn an embedding such that the Euclidean distance is effective. By aggressively max-pooling and downsampling, we are able to construct this embedding using a highly compact neural network. Experiments on a diverse set of multivariate time series datasets show that our approach consistently outperforms existing methods. Measuring distances between examples is a fundamental component of many classification, clustering, segmentation and anomaly detection algorithms for time series BID38 BID43 BID13. Because the distance measure used can have a significant effect on the quality of the , there has been a great deal of work developing effective time series distance measures BID18 BID28 BID1 BID15. Historically, most of these measures have been hand-crafted. However, recent work has shown that a learning approach can often perform better than traditional techniques BID16 BID33 BID9.We introduce a metric learning model for multivariate time series. Specifically, by learning to embed time series in Euclidean space, we obtain a metric that is both highly effective and simple to implement using modern machine learning libraries. Unlike many other deep metric learning approaches for time series, we use a convolutional, rather than a recurrent, neural network, to construct the embedding. This choice, in combination with aggressive maxpooling and downsampling, in a compact, accurate network. Using a convolutional neural network for metric learning per se is not a novel idea BID35 BID45; however, time series present a set of challenges not seen together in other domains, and how best to embed them is far from obvious. In particular, time series suffer from:1. A lack of labeled data. Unlike text or images, time series cannot typically be annotated post-hoc by humans. This has given rise to efforts at unsupervised labeling BID4, and is evidenced by the small size of most labeled time series datasets. Of the 85 datasets in the UCR archive BID10, for example, the largest dataset has fewer than 17000 examples, and many have only a few hundred. 2. A lack of large corpora. In addition to the difficulty of obtaining labels, most researchers have no means of gathering even unlabeled time series at the same scale as images, videos, or text. Even the largest time series corpora, such as those on Physiobank BID19, are tiny compared to the virtually limitless text, image, and video data available on the web. 3. Extraneous data. There is no guarantee that the beginning and end of a time series correspond to the beginning and end of any meaningful phenomenon. I.e., examples of the class or pattern of interest may take place in only a small interval within a much longer time series. The rest of the time series may be noise or transient phenomena between meaningful events BID37 BID21.4. Need for high speed. One consequence of the presence of extraneous data is that many time series algorithms compute distances using every window of data within a time series BID34 BID4 BID37. A time series of length T has O(T) windows of a given length, so it is essential that the operations done at each window be efficient. As a of these challenges, an effective time series distance metric must exhibit the following properties:• Efficiency: Distance measurement must be fast, in terms of both training time and inference time.• Simplicity: As evidenced by the continued dominance of the Dynamic Time Warping (DTW) distance BID42 in the presence of more accurate but more complicated rivals, a distance measure must be simple to understand and implement.• Accuracy: Given a labeled dataset, the metric should yield a smaller distance between similarly labeled time series. This behavior should hold even for small training sets. Our primary contribution is a time series metric learning method, Jiffy, that exhibits all of these properties: it is fast at both training and inference time, simple to understand and implement, and consistently outperforms existing methods across a variety of datasets. We introduce the problem statement and the requisite definitions in Section 2. We summarize existing state-of-the-art approaches (both neural and non-neural) in Section 3 and go on to detail our own approach in Section 4. We then present our in Section 5. The paper concludes with implications of our work and avenues for further research. We first define relevant terms, frame the problem, and state our assumptions. Definition 2.2. Distance Metric A distance metric is defined a distance function d: S × S → R over a set of objects S such that, for any x, y ∈ S, the following properties hold: DISPLAYFORM0 • Identity of Indiscernibles: DISPLAYFORM1 Our approach to learning a metric is to first learn an embedding into a fixed-size vector space, and then use the Euclidean distance on the embedded vectors to measure similarity. Formally, we learn a function f: T D → R N and compute the distance between time series X, Y ∈ T D as: DISPLAYFORM2 2.1 ASSUMPTIONS Jiffy depends on two assumptions about the time series being embedded. First, we assume that all time series are primarily "explained" by one class. This means that we do not consider multilabel tasks or tasks wherein only a small subsequence within each time series is associated with a particular label, while the rest is noise or phenomena for which we have no class label. This assumption is implicitly made by most existing work and is satisfied whenever one has recordings of individual phenomena, such as gestures, heartbeats, or actions. The second assumption is that the time series dataset is not too small, in terms of either number of time series or their lengths. Specifically, we do not consider datasets in which the longest time series is of length T < 40 or the number of examples per class is less than 25. The former number is the smallest number such that our embedding will not be longer than the input in the univariate case, while the latter is the smallest number found in any of our experimental datasets (and therefore the smallest on which we can claim reasonable performance).For datasets too small to satisfy these constraints, we recommend using a traditional distance measure, such as Dynamic Time Warping, that does not rely on a learning phase.3 RELATED WORK Historically, most work on distance measures between time series has consisted of hand-crafted algorithms designed to reflect prior knowledge about the nature of time series. By far the most prevalent is the Dynamic Time Warping (DTW) distance BID42. This is obtained by first aligning two time series using dynamic programming, and then computing the Euclidean distance between them. DTW requires time quadratic in the time series' length in the worst case, but is effectively linear time when used for similarity search; this is thanks to numerous lower bounds that allow early abandoning of the computation in almost all cases BID38.Other handcrafted measures include the Uniform Scaling Distance BID32, the Scaled Warped Matching Distance BID17, the Complexity-Invariant Distance BID2, the Shotgun Distance BID43, and many variants of DTW, such as weighted DTW , DTW-A BID47, and global alignment kernels BID12. However, nearly all of these measures are defined only for univariate time series, and generalizing them to multivariate time series is not trivial BID47. In addition to hand-crafted functions of raw time series, there are numerous hand-crafted representations of time series. Perhaps the most common are Symbolic Aggregate Approximation (SAX) BID32 and its derivatives BID8 BID46. These are discretization techniques that low-pass filter, downsample, and quantize the time series so that they can be treated as strings. Slightly less lossy are Adaptive Piecewise Constant Approximation BID26, Piecewise Aggregate Approximation BID27, and related methods, which approximate time series as sequences of low-order polynomials. The most effective of these representations tend to be extremely complicated; the current state-ofthe-art BID44, for example, entails windowing, Fourier transformation, quantization, bigram extraction, and ANOVA F-tests, among other steps. Moreover, it is not obvious how to generalize them to multivariate time series. A promising alternative to hand-crafted representations and distance functions for time series is metric learning. This can take the form of either learning a distance function directly or learning a representation that can be used with an existing distance function. Among the most well-known methods in the former category is that of BID40, which uses an iterative search to learn data-dependent constraints on DTW alignments. More recently, BID33 use a learned Mahalanobis distance to improve the accuracy of DTW. Both of these approaches yield only a pseudometric, which does not obey the triangle inequality. To come closer to a true metric, BID9 combined a large-margin classification objective with a sampling step (even at test time) to create a DTW-like distance that obeys the triangle inequality with high probability as the sample size increases. In the second category are various works that learn to embed time series into Euclidean space. BID36 use recurrent neural networks in a Siamese architecture BID7 to learn an embedding; they optimize the embeddings to have positive inner products for time series of the same class but negative inner products for those of different classes. A similar approach that does not require class labels is that of BID0. This method trains a Siamese, single-layer CNN to embed time series in a space such that the pairwise Euclidean distances approximate the pairwise DTW distances. BID30 optimize a similar objective, but do so by sampling the pairwise distances and using matrix factorization to directly construct feature representations for the training set (i.e., with no model that could be applied to a separate test set).These methods seek to solve much the same problem as Jiffy but, as we show experimentally, produce metrics of much lower quality. We learn a metric by learning to embed time series into a vector space and comparing the ing vectors with the Euclidean distance. Our embedding function is takes the form of a convolutional neural network, shown in FIG1. The architecture rests on three basic layers: a convolutional layer, maxpooling layer, and a fully connected layer. The convolutional layer is included to learn the appropriate subsequences from the input. The network employs one-dimensional filters convolved over all time steps, in contrast to traditional twodimensional filters used with images. We opt for one-dimensional filters because time series data is characterized by infrequent sampling. Convolving over each of the variables at a given timestep has little intuitive meaning in developing an embedding when each step measurement has no coherent connection to time. For discussion regarding the mathematical connection between a learned convolutional filter and traditional subsequence-based analysis of time series, we direct the reader to BID11.The maxpooling layer allows the network to be resilient to translational noise in the input time series. Unlike most existing neural network architectures, the windows over which we max pool are defined as percentages of the input length, not as constants. This level of pooling allows us to heavily downsample and denoise the input signal and is fed into the final fully connected layer. We downsample heavily after the filters are applied such that each time series is reduced to a fixed size. We do so primarily for efficiency-further discussion on parameter choice for Jiffy may be found in Section 6.We then train the network by appending a softmax layer and using cross-entropy loss with the ADAM BID29 optimizer. We experimented with more traditional metric learning loss functions, rather than a classification objective, but found that they made little or no difference while adding to the complexity of the training procedure; specific loss functions tested include several variations of Siamese networks BID7 BID36 and the triplet loss BID22. For ease of comparison to more traditional distance measures, such as DTW, we present an analysis of Jiffy's complexity. Let T be the length of the D-variable time series being embedded, let F be the number of length K filters used in the convolutional layer, and Let L be the size of the final embedding. The time to apply the convolution and ReLU operations is Θ(T DF K). Following the convolutional layer, the maxpooling and downsampling require (T2DF) time if implemented naively, but (TDF) if an intelligent sliding max function is used, such as that of BID31. Finally, the fully connected layer, which constitutes the embedding, requires Θ(T DF L) time. The total time to generate the embedding is therefore Θ(T DF (K + L)). Given the embeddings, computing the distance between two time series requires Θ(L) time. Note that T no longer appears in either expression thanks to the max pooling. With F = 16, K = 5, L = 40, this computation is dominated by the fully connected layer. Consequently, when L T and embeddings can be generated ahead of time, this enables a significant speedup compared to operating on the original data. Such a situation would arise, e.g., when performing a similarity search between a new query and a fixed or slow-changing database . When both embeddings must be computed on-the-fly, our method is likely to be slower than DTW and other traditional approaches. Before describing our experiments, we first note that, to ensure easy reproduction and extension of our work, all of our code is freely available.1 All of the datasets used are public, and we provide code to clean and operate on them. We evaluate Jiffy-produced embeddings through the task of 1-nearest-neighbor classification, which assesses the extent to which time series sharing the same label tend to be nearby in the embedded space. We choose this task because it is the most widely used benchmark for time series distance and similarity measures BID15 BID1. To enable direct comparison to existing methods, we benchmark Jiffy using datasets employed by BID33. These datasets are taken from various domains and exhibit high variability in the numbers of classes, examples, and variables. We briefly describe each dataset below, and summarize statistics about each in TAB0. • ECG: Electrical recordings of normal and abnormal heartbeats, as measured by two electrodes on the patients' chests.• Wafer: Sensor data collected during the manufacture of semiconductor microelectronics, where the time series are labeled as normal or abnormal.• AUSLAN: Hand and finger positions during the performance of various signs in Australian Sign Language, measured via instrumented gloves.• Trajectories: Recordings of pen (x,y) position and force application as different English characters are written with a pen.• Libras: Hand and arm positions during the performance of various signs in Brazilian Sign Language, extracted from videos.• ArabicDigits: Audio signals produced by utterances of Arabic digits, represented by MelFrequency Cepstral Coefficients. We compare to recent approaches to time series metric learning, as well as popular means of generalizing DTW to the multivariate case:1. MDDTW BID33 ) -MDDTW compares time series using a combination of DTW and the Mahalanobis distance. It learns the precision matrix for the latter using a triplet loss. 2. Siamese RNN BID36 ) -The Siamese RNN feeds each time series through a recurrent neural network and uses the hidden unit activations as the embedding. It trains by feeding pairs of time series through two copies of the network and computing errors based on their inner products in the embedded space. 3. Siamese CNN The Siamese CNN is similar to the Siamese RNN, but uses convolutional, rather than recurrent, neural networks. This approach has proven successful across several computer vision tasks BID7 ). 4. DTW-I, DTW-D -As pointed out by BID47, there are two straightforward ways to generalize DTW to multivariate time series. The first is to treat the time series as D independent sequences of scalars (DTW-I). In this case, one computes the DTW distance for each sequence separately, then sums the . The second option is to treat the time series as one sequence of vectors (DTW-D). In this case, one runs DTW a single time, with elementwise distances equal to the squared Euclidean distances between the D-dimensional elements. 5. Zero Padding -One means of obtaining a fixed-size vector representation of a multivariate time series is to zero-pad such that all time series are the same length, and then treat the "flattened" representation as a vector. 6. Upsampling -Like Zero Padding, but upsamples to the length of the longest time series rather than appending zeros. This approach is known to be effective for univariate time series BID41 ). As shown in TAB1, we match or exceed the performance of all comparison methods on each of the six datasets. Although it is not possible to claim statistical significance in the absence of more datasets (see BID14), the average rank of our method compared to others is higher than its closest competitors at 1.16. The closest second, DTW-I, has an average rank of 3.33 over these six datasets. Not only does Jiffy attain higher classification accuracies than competing methods, but the method also remains consistent in its performance across datasets. This can most easily be seen through the standard deviation in classification accuracies across datasets for each method. Jiffy's standard deviation in accuracy (0.026) is approximately a third of DTWI's (0.071). The closest method in terms of variance is MDDTW with a standard deviation of 0.042, which exhibits a much lower rank than our method. This consistency suggests that Jiffy generalizes well across domains, and would likely remain effective on other datasets not tested here. A natural question when considering the performance of a neural network is whether, or to what extent, the hyperparameters must be modified to achieve good performance on a new dataset. In BID10, and evaluating how classification accuracy varies.6.1 EMBEDDING SIZE Figure 2.left shows that even a few dozen neurons are sufficient to achieve peak accuracy. As a , an embedding layer of 40 neurons is sufficient and leads to an architecture that is compact enough to run on a personal laptop. Figure 2: Effect of fully connected layer size and degree of max pooling on model accuracy using held-out datasets. Even small fully connected layers and large amounts of max poolingup to half of the length of the time series in some cases-have little or no effect on accuracy. For ease of visualization, each dataset's accuracies are scaled such that the largest value is 1.0. The typical assumption in machine learning literature is that max pooling windows in convolutional architectures should be small to limit information loss. In contrast, time series algorithms often max pool globally across each example (e.g. BID20). Contrary to the implicit assumptions of both, we find that the level of pooling that in the highest classification often falls in the 10-25% range, as shown by Figure 2.right We present Jiffy, a simple and efficient metric learning approach to measuring multivariate time series similarity. We show that our method learns a metric that leads to consistent and accurate classification across a diverse range of multivariate time series. Jiffy's resilience to hyperparameter choices and consistent performance across domains provide strong evidence for its utility on a wide range of time series datasets. Future work includes the extension of this approach to multi-label classification and unsupervised learning. There is also potential to further increase Jiffy's speed by replacing the fully connected layer with a structured BID6 or binarized BID39

represent the short summary for finding the corresponding scientific article (long)!

Jiffy is a convolutional approach to learning a distance metric  for multivariate time series that outperforms existing methods in terms of nearest-neighbor classification accuracy.

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.