Question: Represent the following document

Question: We explore the idea of compositional set embeddings that can be used to infer not just a single class, but the set of classes associated with the input data (e.g., image, video, audio signal). This can be useful, for example, in multi-object detection in images, or multi-speaker diarization (one-shot learning) in audio. In particular, we devise and implement two novel models consisting of an embedding function f trained jointly with a “composite” function g that computes set union opera- tions between the classes encoded in two embedding vectors; and embedding f trained jointly with a “query” function h that computes whether the classes en- coded in one embedding subsume the classes encoded in another embedding. In contrast to prior work, these models must both perceive the classes associated with the input examples, and also encode the relationships between different class label sets. In experiments conducted on simulated data, OmniGlot, and COCO datasets, the proposed composite embedding models outperform baselines based on traditional embedding approaches. Embeddings, especially as enabled by advances in deep learning, have found widespread use in natural language processing, object recognition, face identification and verification, speaker verification and diarization (i.e., who is speaking when ), and other areas. What embedding functions have in common is that they map their input into a fixed-length distributed representation (i.e., continuous space) that facilitates more efficient and accurate downstream analysis than simplistic representations such as one-of-k. Moreover, they are amenable to one-shot and few-shot learning since the set of classes that can be represented does not depend directly on the dimensionality of the embedding space. Previous research on embeddings has focused on cases where each example is associated with just one class (e.g., the image contains only one person's face). In contrast, we investigate the case where each example is associated with not just one, but an entire subset of classes from a universe S. The goal is to embed each example so that questions of two types can be answered (see Figure 1 (a)): Is the set of classes in example x a equal to the union of the classes in examples x b and x c? Does the set of classes in example x a subsume the set of classes in example x b? Importantly, we focus on settings in which the classes present in the example must be perceived automatically. We approach this problem using compositional set embeddings. Like traditional embeddings, we train a function f that maps each example x ∈ R n into an embedding space R m so that examples with the same classes are mapped close together and examples with different classes are mapped far apart. Unlike traditional embeddings, our function f is trained to represent the set of classes that is associated with each example, so that questions about set union and subsumption can be answered by comparing vectors in the embedding space. We do not assume that the mechanism by which examples (e.g., images, audio signals) are rendered from multiple classes is known. Rather, the rendering process must be learned from training data. We propose two models, whereby f is trained jointly with either a "composition" function g (Model I) that answers questions about set union, or a "query" function h (Model II) that answers question about subsumption (see Figure 1(a) ). Figure 1: (a): Overview of the paper: embedding function f is trained jointly with either the composition function g or the query function h. In particular, the goal is for g to "compose" the embeddings of two examples, containing classes T and U respectively, to approximate the embedding of an example containing classes T ∪ U. (b): 2-D projection of the embedding space from Experiment 1 on test classes and examples not seen during training (one-shot learning). Function g composes two embeddings (two arrow tails) and maps the back into the embedding space (arrow head). To substantial if imperfect degree, the embedding space is compositional as described in (a). To our knowledge, this computational problem is novel. We see at least two use-cases: Speaker recognition and diarization (i.e., infer who is talking within an audio signal) with multiple simultaneous speakers: Given an audio signal containing speakers who were not part of the training set and who may be speaking simultaneously, and given one example of each person speaking in isolation (one-shot learning), infer which set of speakers is talking. Multi-object recognition in images: Given just the embedding of an image x a, answer whether x a contains the object(s) in another image x b. Storing just the embeddings but not the pixels could potentially be more space-efficient. Because of the novelty of the problem, it was not obvious to what baselines we should compare. When evaluating our models, we sought to assess the unique contribution of the compositional embedding above and beyond what a "traditional" embedding could achieve. Hence, we created baselines by endowing a traditional embedding with some extra functionality to enable it to infer label sets. Modeling assumptions and notation: For generality, we refer to the data to be embedded (images, videos, audio signals, etc.) simply as "examples". Let the universe of classes be S. From any subset T ⊆ S, a ground-truth rendering function r: 2 S → R n "renders" an example, i.e., r(T) = x. Inversely, there is also a ground-truth classification function c: R n → 2 S that identifies the label set from the rendered example, i.e., c(x) = T. Neither r nor c is observed. We let e T represent the embedding (i.e., output of f) associated with some example containing classes T. Contribution: To our knowledge, this is the first paper to explore how embedding functions can be trained both to perceive multiple objects in the example and to represent the set of detected objects so that set operations can be conducted among embedded vectors. We instantiate this idea in two ways: Model I for set union (f & g) and Model II for set containment (f & h). By evaluating on synthetic data, OmniGlot handwritten image data , as well as the COCO dataset , we provide a proof-of-concept that "compositional set embeddings" can work. Embeddings: We distinguish between two types of embeddings: "Perceptual" embeddings such as for vision (Facenet ) and speech (x-vector ) where each class (e.g., person whose voice was recorded or face was photographed) may contain widely varying examples across speech content, facial expression, lighting, noise, etc. Word embeddings (word2vec , GloVe ) where each class contains only one exemplar by definition. Within the former, the task of the embedding function is to map examples from the same class close together and examples from other classes far apart. This often requires deep, non-linear transformations to be successful. With word embeddings, the class of each example is already clear and does not need to be inferred; instead, the goal is to give the embedded vectors geometric structure to reflect co-occurrence, similarity in meaning, etc. Compositional embeddings: Since at least 30 years, AI researchers, cognitive scientists, and computational neuroscientists have explored how the embeddings of multiple elements could be combined to reflect relationships between them or higher-level semantics. However, almost all this work was based on word embeddings, in which perception was not necessary. Some early work investigated how the grammatical structure and/or semantics of an input sentence can be represented in an efficient manner in neural networks, and how such a network could be trained . Given the advent of word embeddings, deep NLP architectures can combine the word-level semantics, as represented by the embeddings of the individual elements of an input sentence, to infer higher-level attributes, e.g., sentiment . Recent work has investigated to what extent contemporary recurrent neural networks can generalize to understand novel sentences consisting of known words. Also, in the NLP domain, developed compositional pairwise embeddings that model the co-occurrence relationships between two words given their common context. Probably the most algorithmically similar work to ours is by on compositional network embeddings: the goal is to predict whether two new nodes in a graph, which were not observed during training, are adjacent, using node-based features as predictors. In their approach, two embeddings are used: one to embed the node-based features, and another to aggregate these embedded features into a secondary embedding space. Structurally, their work differs from ours in that the two embedding spaces in their model do not represent the same universe of objects; the embeddings do not capture set relationships. Deep set representations: Our paper is also about how to encode a set of objects with a neural network. One issue is how to ensure invariance to the order in which examples are presented. proposed an approach based on permutation-invariant content-based attention. For producing sets as outputs, proposed a probabilistic model, within a supervised learning paradigm where all classes are known at training time, that predicts both the cardinality and the particular elements of the set. Given two examples x a and x b that are associated with singleton sets {s} and {t}, respectively, the hope is that, for some third example x c that is associated with both classes (i.e., {s, t}), we have Moreover, we hope that g can generalize to any number of classes within the set S. For example, if example x d is associated with a singleton set {u}, then we hope where x e is an example associated with {s, t, u}. There are two challenging tasks that f and g must solve cooperatively: f has to learn to perceive multiple objects that appear simultaneously and may possibly interact with each other -all without knowing the rendering process r of how examples are formed or how classes are combined. g has to define geometrical structure in the embedding space to support set union operations. One way to understand our computational problem is the following: If f is invertible, then ideally we would want g to compute g(e T, e U) = f (r(c(f −1 (e T)) ∪ c(f −1 (e U)))). In other words, one (though not necessarily the only) way that g can perform well is to learn to perform the following actions (without knowing r or c): invert each of the two input embeddings; classify the two corresponding label sets; render an example with the union of the two inferred label sets; and embed the . One-shot learning: Model I can be used for one-shot learning on a set of classes S not seen during training in the following way: We obtain k labeled examples x 1,..., x k from the user, where each {s i} = c(x i) is the singleton set formed from the ith element of S and |S| = k. We call these examples the reference examples. We then infer which set of classes is represented by a new example x using the following procedure: Compute the embedding of x, i.e., f (x). Use f to compute the embedding of each singleton example x i, i.e., e {i} = f (x i). From e {1},..., e {k}, estimate the embedding of every subset T = {s 1, . . ., s l} ⊆ S according to the recurrence relation: Finally, estimate the label of x as arg min Although the number of possible subsets is exponential in |S|, for speaker diarization the number of overlapping speakers is typically small, and thus the iteration is tractable. Functions f and g are trained jointly: For each example x associated with classes T, we compute e T from the singleton prototypes according to Equation 1. (To decide the order in which we apply the recursion, we define an arbitrary ordering over the elements of S and iterate accordingly.) We then compute a hinge loss: where is a small positive real number. In practice, for each example x, we randomly pick one element of T ∈ 2 S for comparison. See Appendix for a discussion of an alternative (but less effective) training procedure. To explore the viability of Model I, we first conducted a simulation using 1-D "audio" signals, in which each "speaker" s i ∈ S is modeled with a prototype p i ∈ R n consisting of the superposition of some randomly chosen frequencies and phases. Then, the simultaneous sound from multiple (up to 3) speakers is given by a rendering function r that computes a vector sum of slightly perturbed versions of the prototype vectors and then clips the . (See Figure 2 (a) for example waveforms, and the Appendix for more details.) We trained an embedding and composition functions f and g, as described in Section 3.1, on a set of 250 speakers. We then tested these functions on test speakers (and examples) not seen during training after providing just a single "reference" example of each speaker for one-shot learning. The goal is to identify the exact set of speakers who contributed to the formation of each audio waveform. Architecture: For function f we used a simple convolutional neural network that produces unitlength vectors in R 32: where Conv(k, f) is a 1-D convolutional layer with f filters of size k, stride 1, and zero-padding to preserve the spatial extent of the example; BN is batch normalization; MP(k) is a max-pooling layer with width and stride k; FC(n) is a fully-connected layer with n neurons; and L2Norm is a L 2 -normalization layer. We constructed the composition function (g DNN) to be symmetric: is a symmetric function (with parameter matrices W 1, W 2) of its two examples a, b ∈ R n that produces a vector in R k. Assessment: We evaluated accuracy (% correct) in multiple ways: the accuracy, over all examples, of identifying the exact set of speakers; the accuracy, separately for each set size, of identifying the exact set of speakers; and the accuracy, over all examples, in distinguishing just the number of speakers in the set rather than their specific identities. As a baseline, we used an oracle that returned the Most Frequently (MF) occurring label set in the test set (or a random selection among multiple most frequent elements). Results are shown in Figure 2 (b). Note that, for the rows that evaluate on k-sets, the embedding functions are still used to compare the test example to every possible subset (see Equation 2) -not just subsets of size k. In general, the compositional embedding method was far more accurate (48.1% for top-1 and 69.6% for top-3 on all subsets) compared to the baseline. The accuracy improvement was largest on singletons (1-sets), which made up 1/3 of the test data, and for which any non-compositional embedding functions (i.e., f trained without any g) would likely work well. Even on 2-sets and 3-sets, however, the compositional embedding approach was substantially more accurate than the baseline. Moreover, part of the task involves inferring the number of elements in the label set (set size determination); on this problem too the compositional embedding was nearly twice as accurate (61.9% vs. 33.3%) compared to the MF baseline. Geometry of the embedding space: We visualized (using PCA) the embedding space for a set of 3 classes not used for training (Figure 1(b) ). Each marker (the symbol/color combination distinguishes the different label sets) represents the embedding of a test example. The clusters of markers for each label set are clearly distinguished by the embedding f. From the examples from each label set, one is randomly chosen as the "reference" examples for 1-shot learning. Using composition function g DNN, their embeddings are combined to estimate where, in the embedding space, an example for the union of their two label sets would lie; the estimated location is shown at the head of the arrow whose two tails come from the two reference examples. Although the estimated and actual clusters do not align exactly, there is substantial agreement. For instance, the examples for speakers {1, 2, 3} (i.e., all 3 people talking simultaneously) are represented by the blue triangles pointing to the right, and the estimate, given by f and g DNN from reference examples of {3} and {1, 2} is shown at the head of the arrow with label {1, 2, 3} beside it. Not all compositions are accurate, e.g., the estimated location for label set {1, 3} is somewhat below the actual cluster location. We also evaluated our method on the OmniGlot dataset . OmniGlot contains handwritten characters from 50 different alphabets; in total it comprises 1623 symbols, each of which was drawn by 20 people and rendered as a 64x64 image. OmniGlot has been previously used in one-shot learning research (e.g., ;). In our experiment, the model is provided with one reference image for each singleton test class. Then, it uses f and g to select the subset of classes that most closely match the embedding of each test example (Equation 2). In this study we considered class label sets up to size 2 (i.e., singletons and 2-sets). The rendering function r randomly picks one of the 20 exemplars from each class; it randomly shifts, scales, and rotates it; and then it adds it to an image with Gaussian noise (see Figure 3(a) ). For images with multiple classes, the pixel-wise minimum across all classes is applied before adding the noise. (See Appendix for more details.) Similar to Experiment 1, the goal is to train f and g so that, on classes not seen during training, the exact set of classes contained in each test example can be inferred. During both training and testing, each of the 15 class label sets (5 singletons and 5 2 = 10 2-sets) occurred with equal frequency. All the embeddings were normalized to unit length. Architecture: For f, we used ResNet-18 Training: The process is similar to Experiment 1. See Appendix for details. Testing: Similar to training, the testing data are grouped into 5 randomly selected classes (not seen during training), and images from these classes are rendered using function r from either singleton or 2-set class label sets. We optimize Equation 2 to estimate the label set for each test example. Baselines: We compared to several baselines: • Most frequent (MF): Always guess the most frequent element in the test set. Since all classes occurred equally frequently, this was equivalent to random guessing. • Traditional f with simulated composite reference examples (SimRef): Given the reference examples of the singleton label sets (e.g., {1}, {3}), we can simulate reference examples for each composite label set (e.g., {1, 3}) and then use a traditional embedding approach to classify the test examples. We simulated imperfect knowledge of the rendering function r by performing pixel-wise minimum, but without shifting/scaling/rotation. In other words, this approach simply superimposes pairs of reference images on top of each other to simulate composite examples. To estimate the label set of each test example, we select the reference example (including the simulated composite references) with minimal L 2 distance. • Traditional f and average (Mean): f is trained as a traditional embedding function (only on singletons) using one-shot learning . Then the embedding of a composite image is computed using the mean of the embeddings of each class in its label set. • Traditional f trained jointly with g mean (g mean): f is trained on only singleton embeddings but is optimized jointly with composition g mean that computes the mean of its inputs. The difference with the previous baseline is that f can benefit from knowing how its embeddings are combined. Results: As shown in Figure 3 (b), the proposed f & g method outperforms the Mean and g mean baselines. Upon closer investigation, we discovered that, while Mean can distinguish each singleton from the other four singletons with high accuracy (0.979, not shown in table), it struggles when it must decide among all 15 possible label sets. The slightly more powerful f & g Mean method (i.e., the composition function has no trainable parameters) can achieve better accuracy on 2-set images, but the accuracy on 1-set images was much worse than other methods. We argue that f and g correspond to two different tasks, and the model is not able to do a good job on both tasks at the same time without optimizing g. If g is a learnable function, then a simple symmetric linear function (g Lin) achieves the best performance. After we stack more FC layers, the performance gets worse. The reason could be overfitting, and it may be possible to achieve better performance with regularization or more training data. In real life, it is very difficult to create a dataset that all singletons in a compositional set are labeled. Thus, we want to extend our compositional embedding from "composition" to "containing". Here we consider a second type of compositional embedding mechanism that tests whether the set of classes associated with one example subsumes the set of classes associated with another example. We implement this using a "query" function h that takes two embedded examples as inputs: In contrast to g, function h is not symmetric: its first and second arguments are the putative superset and subset, respectively. Also, h can be trained in an unsupervised manner w.r.t. the individual examples: it never needs to know which particular label set is associated with an example, but rather only pairwise information about which examples "subsume" other examples. For h, we tried several functions (h DNN, h Lin+FC, h Lin), analogous to the different implementations of g from Section 3.3. The final layers of all models have a 1-dimensional output. Ref. Functions f and h are trained jointly. Since h is not symmetric, its first layer is replaced with two weight matrices for the different input embeddings (see Appendix). In contrast to Model I, "reference" examples are not needed; only the subset relationships between label sets of pairs of examples are required. Model II can be trained on one set of classes and applied to a different set of classes (see Experiment 3), akin to zero-shot learning. It is also useful when the training and testing classes are the same because, in contrast to traditional supervised training of a detector for each possible class, no labels for individual examples are needed. To train f and h, we backpropagate a binary cross-entropy loss, based on correctly answering the query in Eq. 3, through both f and h. Here we assess Model II in a one-shot learning setting (OmniGlot), i.e., training and testing classes are different. As in Experiment 2, we consider class label sets of size up to 2, and we use the same rendering function r. Let f (x a) and f (x b) be the second arguments to h. For x a, we always choose examples associated with two classes, i.e., c(x a) = {s 1, s 2}. For x b, half of the positive examples (i.e., such that h(f (x a), f (x b)) = True) contain either s 1 or s 2, and half contain both classes. For the negative examples (h(f (x a), f (x b)) = False), x b is associated with some other singleton class s 3 ∈ {s 1, s 2}. See Appendix for more details. Baseline: We compared our proposed method with a traditional (non-compositional) embedding method (TradEmb) that is trained to separate examples according to their association with just a single class. In particular, for each composite example x a (i.e., |c(x a)| = 2), we picked one of the two classes arbitrarily (according to some fixed ordering on the elements of S); call this class s 1. Then, we chose both a positive example x b (such that c(x b) = {s 1}) and a negative example x c (such that c(x c) = {s 3} ⊂ c(x a)). We then compute a hinge loss so that the distance between f (x a) and f (x b) is smaller than the distance between f (x a) and f (x c), and backpropagate the loss through f. During testing, we use f to answer a query -does c(x a) contain c(x b)? -by thresholding the distance between f (x a) and f (x b) (threshold of 0.5). Results are shown in Table 1 (a). Compositional embeddings, as implemented with a combination of f trained jointly with either h DNN, h Lin+FC, or h Lin, slightly outperformed the TradEmb baseline, in terms of both % correct accuracy and AUC. Moreover, there was an advantage of deeper architectures for h. We trained and evaluated Model II on COCO . This is a highly challenging problem: in the example in Figure 4, f has to encode a backpack, a cell phone, and a person; then, given completely different images of these classes (and others), h has to decide which objects were present in the original image. Here, training and testing classes are the same (but testing examples are not used for training). In COCO, each image may contain objects from multiple classes, and each object has a bounding box. We used the bounding boxes to crop singleton objects from images. For training, we used the same strategy as Experiment 3, except that in positive queries (i.e., pairs of images x a, x b such that h(f (x a), f (x b)) = True), image x b was always associated with just a singleton label set. During testing, half of x b are contained in x a and half are not (see Appendix). Baseline: We compared to the TradEmb method: to decide the singleton label set for each image during training, we pick the largest object. Since in this problem the training and testing classes are the same, we also implemented another baseline (see Appendix) consisting of a ResNet classifier with multiple independent binary outputs (one for each class). We can then answer queries about the label sets of two images using the labels of the different classes estimated by the ResNet for each image. Results are shown in Table 1 (b). The proposed methods (f & one of the h functions) easily outperformed TradEmb. They also outperformed the ResNet baseline. One possible reason for the latter baseline's poor performance was that the data was highly imbalanced (since most images contain just a few images). As in Experiment 3, deeper architectures for h performed better. We proposed a new kind of embedding mechanism whereby the set of objects contained in the input data (e.g., image, video, audio) must be both perceived and then mapped into a space such that the set relationships -union (Model I) and subset (Model II) -between multiple embedded vectors can be inferred. Importantly, the ground-truth rendering process for how examples are rendered from their component classes is not known and must implicitly be learned. In our experiments, conducted on simulated data, OmniGlot, and COCO, the accuracy was far from perfect but outperformed several baselines, including one based on a traditional embedding approach. The provide a proof-of-concept of how an embedding function f, trained jointly with either the composition function g or the query function h, could be effectively optimized. One possible direction for further research to increase accuracy is to take better advantage of the statistical structure of class co-occurrence in a specific application domain (e.g., which objects tend to co-occur in the same image). A ALTERNATIVE TRAINING PROCEDURE We also tried another method of training f and g with the explicit goal of encouraging g to map e T and e U to be close to e T ∪U. This can be done by training f and g alternately, or by training them jointly in the same backpropagation. However, this approach yielded very poor . A possible explanation is that g could fulfill its goal by mapping all vectors to the same location (e.g., 0). Hence, a trade-off arises between g's goal and f's goal (separating examples with distinct label sets). Since it was initially not clear to us whether functions f and g even exist with the desired properties that could plausibly be implemented as a neural network, we first sought to design them analytically. Composition function g needs to map two vectors on a unit sphere back onto the same sphere, so that the location on the sphere uniquely identifies a set of classes. Here, we assume the set of classes is the same for training and testing. Our construction is inspired by basic group theory, in particular how each element of a permutation group can be represented as a rotation of Euclidean space and performed by multiplication with a permutation matrix. Let |S| = k. We model g after the permutation group of degree m = 2k consisting of all pairwise exchanges of coordinates 2i and 2i − 1 (i = 1, . . ., k), where the group action is composition. Since each such permutation is independent of all the others, the group is commutative and contains contains 2 k elements -just as we desire so that the range of g contains 2 |S| elements. Each member of the permutation group can be associated with a unique permutation matrix, where multiplication among these matrices is also commutative. Define a vector e ∅ ∈ R 2k whose length is 1 and whose components are all distinct. The first condition is standard when creating an embedding, and the second is important so that the permutations applied to the vector can be deduced unambiguously. We associate vector e ∅ with the empty set ∅. Next, to each singleton {s i}, where s i ∈ S, we associate the m-dimensional permutation matrix P {si} that swaps axes 2i and 2i − 1. We define f as a neural network: In its first layers, all the classes associated with example x are detected independently using k binary classifiers. Next, for all the detected classes, their associated permutation matrices P are multiplied together, which yields another commutative permutation matrix -the ant matrix identifies all the classes present in the example. Finally, the is multiplied by vector e ∅. This produces a vector with unit length in the embedding space R m. If the first few layers comprise k binary classifiers d i (z) ∈ {0, 1} (i = 1, . . ., m), then the whole f network computes: We then construct g so that, given two embedded vectors e T, e U, it computes the product of their associated permutation matrices and multiplies the by e ∅: g(e T, e U) = g(P T e ∅, P U e ∅). This construction of f and g enables perfect inference of the union of classes associated with any examples, provided the binary classifiers are perfect. This gave us hope that compositional set embeddings could succeed. Each speaker was represented by a "prototype" waveform comprising the sum of 8 sinusoids (range [−1, +1]) with randomly chosen frequencies and phases. To construct each training and testing example, the prototypes of all the speakers contained in the particular example are added together and then clipped at [−4, +4]. To simulate noise, we slightly perturbed the frequencies and phases of each prototype by random noise drawn uniformly from [−0.02, +0.02]. We also perturbed the superposition by a small amount (from [−0.02, +0.02]) prior to clipping. The training dataset contained a total of 250 simulated speakers. Each minibatch comprised 3072 examples from 5 unique speakers that were combined in randomly selected subsets T: 1024 of these contained individual speakers (singletons), 1024 contained combinations of 2 speakers, and 1024 contained combinations of 3 speakers. Functions f and g were trained jointly as described in Section 3.1. Training was conducted for 4000 epochs with a learning rate of 0.002 and the Adam algorithm. We set the hinge parameter = 0.1. At test time, one of the singleton examples was randomly chosen as the "reference" example for one-shot learning. Similar to training, each test set consists of 3072 examples comprising randomly chosen 1-sets, 2-sets and 3-sets of speakers; see Figure 2 (a) for example waveforms. To generate an example image from a class prototype, we applied random affine transformations consisting of shift up to 20%, scaling up to 10%, and rotation up to 10 •. Gaussian noise was added with mean 0.9 and variance 0.1. We set the hinge parameter = 0.1. Training is performed on 964 OmniGlot classes; from each of these, the 20 images are augmented using affine transformations (described above) to yield 50 images. Each mini-batch comprises 10000 examples from 5 classes, with two images randomly selected per class: one as a reference example, and one as a training example. The validation set consists of 20 OmniGlot classes not used for training. Training is performed using Adam (lr = .0003) to maximize the validation accuracy, up to a maximum of 100 epochs. Testing accuracy is computed over the 659 OmniGlot classes not used for training or validation. The dataset used in this experiment is the same as experiment 2 including the distribution of training set, validation set and test set. We also used the same ResNet-18 as for Experiment 2. Function h was constructed the same as g except that the first layer was relaxed to be asymmetric: W 1 e T + W 2 e U, and the output layer is logistic sigmoid. For training and testing, the proportion of positive and negative queries was fixed to be equal. During training, each mini-batch contains 128 query pairs and each query pair is composed by a positive query and a negative query. We use binary cross entropy loss and Adam optimizer (lr = .0003) to optimize the models. During evaluation, 0.5 is used for thresholding. COCO's training set is used for training, and COCO's labels of subclasses are used as training labels. A small part of images are taken as validation set. For ResNet baseline, We modified ResNet's last's dimension to 80 and applied logistic sigmoid to all of them. Thus, the output layer can be used as 80 classifiers and each one corresponds to one possible subclass. During training, the loss is composed by binary cross entropy loss from all 80 binary classifiers. For every subclass in all 80 possibilities, if it is in the set of subclasses of one image, then the label is 1, otherwise 0. Adam optimizer ((lr) =.0003) is used for optimization. During evaluation, for each query we assume the class labels of subclass images are already known. ResNet is not optimized to answer queries about image pairs. Instead, it tries to encode each image into an n-bit string (for n classes). While this representation can account for all 2 n possible label sets, it may not be the most effective or efficient representation for the task, especially since some objects are very unlikely to co-occur with others. The proposed f &h embedding method can harness the co-occurrence structure to answer queries more accurately, whereas a ResNet trained to recognize all the individual classes does not harness it. For other models singleton images are also required, which are cropped according to COCO's labels of bounding boxes. All images were padded with zeros to be square and then downscaled to 128x128. We used the same architecture as for Experiment 3, except that the input layer was 3 channels instead of 1. Testing was the same as Experiment 3 (likewise with a distance threshold of 0.5). In both training and testing, the number of positive queries and negative queries are balanced. In order to explore how the number of subclasses affects h function, we also plot the accuracy and AUC with different number of subclasses in Figure 5. We can see a decreasing trend of performance when the images contain more kinds of labeled objects.

We explored how a novel method of compositional set embeddings can both perceive and represent not just a single class but an entire set of classes that is associated with the input data.

Represent the TLDR for finding the corresponding scientific article

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: In this work, we propose a goal-driven collaborative task that contains language, vision, and action in a virtual environment as its core components. Specifically, we develop a Collaborative image-Drawing game between two agents, called CoDraw. Our game is grounded in a virtual world that contains movable clip art objects. The game involves two players: a Teller and a Drawer. The Teller sees an abstract scene containing multiple clip art pieces in a semantically meaningful configuration, while the Drawer tries to reconstruct the scene on an empty canvas using available clip art pieces. The two players communicate via two-way communication using natural language. We collect the CoDraw dataset of ~10K dialogs consisting of ~138K messages exchanged between human agents. We define protocols and metrics to evaluate the effectiveness of learned agents on this testbed, highlighting the need for a novel "crosstalk" condition which pairs agents trained independently on disjoint subsets of the training data for evaluation. We present models for our task, including simple but effective baselines and neural network approaches trained using a combination of imitation learning and goal-driven training. All models are benchmarked using both fully automated evaluation and by playing the game with live human agents. Building agents that can interact with humans in natural language while perceiving and taking actions in their environments is one of the fundamental goals in artificial intelligence. One of the required components, language understanding, has traditionally been studied in isolation and with tasks aimed at imitating human behavior (e.g. language modeling BID4 ; BID35, machine translation BID2 ; BID42, etc.) by learning from large text-only corpora. To incorporate both vision and action, it is important to have the language grounded BID19 BID3, where words like cat are connected to visual percepts and words like move relate to actions taken in an environment. Additionally, judging language understanding purely based on the ability to mimic human utterances has limitations: there are many ways to express roughly the same meaning, and conveying the correct information is often more important than the particular choice of words. An alternative approach, which has recently gained increased prominence, is to train and evaluate language generation capabilities in an interactive setting, where the focus is on successfully communicating information that an agent must share in order to achieve its goals. In this paper, we propose the Collaborative Drawing (CoDraw) task, which combines grounded language understanding and learning effective goal-driven communication into a single, unified testbed. This task involves perception, communication, and actions in a partially observable virtual environment. As shown in FIG0, our game is grounded in a virtual world constructed by clip art objects. Two players, Teller and Drawer, play the game. The Teller sees an abstract scene made from clip art objects in a semantically meaningful configuration, while the Drawer sees a drawing canvas that is initially empty. The goal of the game is to have both players communicate so that the Drawer can reconstruct the image of the Teller, without ever seeing it. Our task requires effective communication because the two players cannot see each other's scenes. The Teller has to describe the scene in sufficient detail for the Drawer to reconstruct it, which will require rich grounded language. Moreover, the Drawer will need to carry out a series of actions from a rich action space to position, orient, and resize all of the clip art pieces required for the reconstruction. Note that such actions are only made possible through clip art pieces: they can represent semantically meaningful configurations of a visual scene that are easy to manipulate, in contrast to low-level pixel-based image representations. The performance of a pair of agents is judged based on the quality of reconstructed scenes. We consider high-quality reconstructions as a signal that communication has been successful. As we develop models and protocols for CoDraw, we found it critical to train the Teller and the Drawer separately on disjoint subsets of the training data. Otherwise, the two machine agents may conspire to successfully achieve the goal while communicating using a shared "codebook" that bears little resemblance to natural language. We call this separate-training, joint-evaluation protocol crosstalk, which prevents learning of mutually agreed upon codebooks, while still checking for goal completion at test time. We highlight crosstalk as one of our contributions, and believe it can be generally applicable to other related tasks BID41 BID14 BID11 BID9 BID27. • We propose a novel CoDraw task, which is a game designed to facilitate the learning and evaluation of effective natural language communication in a grounded context. • We collect a CoDraw dataset of ∼10K variable-length dialogs consisting of ∼138K messages with the drawing history at each step of the dialog.• We define a scene similarity metric, which allows us to automatically evaluate the effectiveness of the communication at the end and at intermediate states.• We propose a cross-talk training and evaluation protocol that prevents agents from potentially learning joint uninterpretable codebooks, rendering them ineffective at communicating with humans.• We evaluate several Drawer and Teller models automatically as well as by pairing them with humans, and show that long-term planning and context reasoning in the conversation are key challenges of the CoDraw task. Language and Vision. The proposed CoDraw game is related to several well-known language and vision tasks that study grounded language understanding BID21 BID13 BID11. For instance, compared to image captioning BID46 BID6 BID32, visual question answering BID1 BID23 BID33 BID37 BID43 BID54 and recent embodied extensions BID10, CoDraw involves multiple rounds of interactions between two agents. Both agents hold their own partially observable states and need to build a mental model for each other to collaborate. Compared to work on navigation BID47 BID0 BID16 where an agent must follow instructions to move itself in a static environment, CoDraw involves moving and manipulating multiple clip art pieces, which must jointly form a semantically meaningful scene. Compared to visual dialog BID8 BID39 BID40 BID36 tasks, agents need to additionally cooperate to change the environment with actions (e.g., move around pieces). Thus, the agents have to possess the ability to adapt and hold a dialog about novel scenes that will be constructed as a consequence of their dialog. In addition, we also want to highlight that CoDraw has a well-defined communication goal, which facilitates objective measurement of success and enables end-to-end goal-driven learning. End-to-end Goal-Driven Dialog. Traditional goal-driven agents are often based on'slot filling' BID24 BID51 BID54, in which the structure of the dialog is pre-specified but the individual slots are replaced by relevant information. Recently, endto-end neural models are also proposed for goal-driven dialog BID5 BID29 BID39 BID20, as well as goal-free dialog or'chit-chat' BID38 BID39 BID45 BID12. Unlike CoDraw, in these approaches, symbols in the dialog are not grounded into visual objects. Language Grounded in Environments. Learning language games to change the environment has been studied recently BID49. The agent can change the environment using the grounded natural language. However, agents do not have the need to cooperate. Other work on grounded instruction following relies on datasets of pre-generated action sequences annotated with human descriptions, rather than using a single end goal BID31. Speaker models for these tasks are only evaluated based on their ability to describe an action sequence that is given to them BID15, whereas Teller models for CoDraw also need to select a desired action sequence in a goal-driven manner. Language grounding has also been studied for robot navigation, manipulation, and environment mapping BID44 BID34 BID7. However, these works manually pair each command with robot actions and lack end-to-end training BID44, dialog BID34 BID7, or both BID48.Emergent Communication. Building on the seminal works by BID25 BID26, a number of recent works study cooperative games between agents where communication protocols emerge as a consequence of training the agents to accomplish shared goals BID41 BID14. These methods have typically been applied to learn to communicate small amounts of information, rather than the complete, semantically meaningful scenes used in the CoDraw task. In addition, the learned communication protocols are usually not natural BID22 or interpretable. On the other hand, since our goal is to develop agents that can assist and communicate with humans, we must pre-train our agents on human communication and use techniques that can cope with the greater linguistic variety and richness of meaning present in natural language. In this section, we first detail our task, then present the CoDraw dataset, and finally propose a Scene Similarity Metric which allows automatic evaluation of the reconstructed and original scene. Abstract Scenes. To enable people to easily draw semantically rich scenes on a canvas, we leverage the Abstract Scenes dataset of. This dataset consists of 10,020 semantically consistent scenes created by human annotators. An example scene is shown in the left portion of FIG0. Most scenes contain 6 objects (min 6, max 17, mean 6.67). These scenes depict children playing in a park, and are made from a library of 58 clip arts, including a boy (Mike) and a girl (Jenny) in one of 7 poses and 5 expressions, and various other objects including trees, toys, hats, animals, food, etc. An abstract scene is created by dragging and dropping multiple clip art objects to any (x, y) position on the canvas. Also, for each clip art, different spatial transformations can be applied, including sizes (Small, Normal, Large), and two orientations (facing left or right). The clip art serve simultaneously as a high-level visual representation and as a mechanism by which rich drawing actions can be carried out. Interface. We built a drag-and-drop interface based on the Visual Dialog chat interface BID8 ) (see Figures 4 and 5 in Appendix A for screen shots of the interface). The interface allows real-time interaction between two people. During the conversation, the Teller describes the scene and answers any questions from the Drawer on the chat interface, while Drawer "draws" or reconstructs the scene based on the Teller's descriptions and instructions. Each side is only allowed to send one message at a time, and must wait for a reply before continuing. The maximum length of a single message is capped at 140 characters: this prevents excessively verbose descriptions and gives the Drawer more chances to participate in the dialog by encouraging the Teller to pause more frequently. Both participants were asked to submit the task when they are both confident that Drawer has accurately reconstructed the scene of Teller. Our dataset, as well as this infrastructure for live chat with live drawing, will be made publicly available. Additional Interaction. We did not allow Teller to continuously observe Drawer's canvas to make sure that the natural language focused on the high-level semantics of the scene rather than instructions calling for the execution of low-level clip art manipulation actions, but we hypothesize that direct visual feedback may be necessary to get the all the details right. For this, we give one chance for the Teller to look at the Drawer's canvas using a'peek' button in the interface. Communication is only allowed after the peek window is closed. We collect 9,993 1 dialogs where pairs of people complete the CoDraw task, consisting of one dialog per scene in the Abstract Scenes dataset. The dialogs contain of a total of 138K utterances and include snapshots of the intermediate state of the Drawer's canvas after each round of each conversation. We reserve 10% of the scenes to form a test set and an additional 10% to form a development set; the corresponding dialogs are used to evaluate human performance for the CoDraw task. The remaining dialogs are used for training (see Section 5 for details about our training and evaluation setup.)The message length distribution for the Drawer is skewed toward 1 with the passive replies like "ok", "done", etc. There does exist a heavy tail, which shows that Drawers do ask clarifying questions about the scene like "where is trunk of second tree, low or high". On the other hand, the distribution of number of tokens in Tellers' utterances is relatively smooth with long tails. The vocabulary size is 4,555. Since the subject of conversations is about abstract scenes with a limited number of clip arts, the vocabulary is relatively small compared to those on real images. See Appendix B for a more detailed analysis of our dataset, where we study the lengths of the conversations, the number of rounds, and the distributions of scene similarity scores when humans perform the task. The goal-driven nature of our task naturally lends itself to evaluation by measuring the similarity of the reconstructed scene to the original. For this purpose we define a scene similarity metric, which allows us to automatically evaluate communication effectiveness both at the end of a dialog and at intermediate states. We use the metric to compare how well different machine-machine, humanmachine, and human-human pairs can complete the CoDraw task. Let c i, c j denote the identity, location, configuration of two clipart pieces i and j. A clipart image C = {c i} is then simply a set of clipart pieces. Given two images C andĈ, we compute scene similarity by first finding the common clipart pieces C ∩Ĉ and then computing unary f (c i) and pairwise terms g(c i, c j) on these pieces in common: DISPLAYFORM0 Using f (c) = 1 and g(c i, c j) = 0 would in the standard intersection-over-union measure used for scoring set predictions. The denominator terms normalize the metric to penalize missing or extra clip art, and we set f and g such that our metric is on a 0-5 scale. The exact terms f and g are described in Appendix C. We model both the Teller and the Drawer, and evaluate the agents using the metrics described in the previous section. Informed by our analysis of the collected dataset (see Appendix B), we make three modeling assumptions compared to the full generality of the setup that humans were presented with during data collection. These assumptions hold for all models studied in this paper. Assumption 1: Silent Drawer. We choose to omit the Drawer's ability to ask clarification questions; instead, our Drawer models will always answer "ok" and our Teller models will not condition on the text of the Drawer replies. This is consistent with typical human replies (around 62% of which only use a single token) and the fact that the Drawer talking is not strictly required to resolve the information asymmetry inherent in the task. We note that this assumption does not reduce the number of modalities needed to solve the task: there is still language generation on the Teller side, in addition to language understanding, scene perception, and scene generation on the Drawer side. Drawer models that can detect when a clarification is required, and then generate a natural language clarification question is interesting future work. Assumption 2: No Peek Action. The second difference is that the data collection process for humans gives the Teller a single chance to peek at the Drawer's canvas, a behavior we omit from our models. Rich communication is still required without this behavior, and omitting it also does not decrease the number of modalities needed to complete the task. We leave for future work the creation of models that can peek at the time that maximizes task effectiveness. Assumption 3: Full Clip Art Library. The final difference is that our drawer models can select from the full clip art library. Humans are only given access to a smaller set so that it can easily fit in the user interface, while ensuring that all pieces needed to reconstruct the target scene are available. We choose to adopt the full-library condition as the standard for models because it gives the models greater latitude to make mistakes (making the problem more challenging) and makes it easier to detect obviously incorrect groundings. Simple methods can be quite effective even for what appear to be challenging tasks, so we begin by building models based on nearest-neighbors and rule-based approaches. Rule-based Nearest-Neighbor Teller. For our first Teller model, We consider a rule-based dialog policy where the Teller describes exactly one clip art each time it talks. The rule-based system determines which clip art to describe during each round of conversation, following a fixed order that roughly starts with objects in the sky (sun, clouds, airplanes), then objects in the scene (trees, Mike, Jenny), and ends with small objects (sunglasses, baseball bat). The Teller then produces an utterance by performing a nearest-neighbor lookup in a database containing (Teller utterance, clip art object) pairs, where the similarity between the selected clip art and each database element is measured by applying the scene similarity metric to individual clip art. The database is constructed by collecting all instances in the training data where the Teller sent a message and the Drawer responded by adding a single clip art piece to the canvas. Instances where the Drawer added multiple clip art pieces or made any changes to the position or other attributes of pieces already on the canvas are not used when constructing the nearest-neighbor database. This baseline approach is based on the assumptions that the Drawer's action was elicited by the Teller utterance immediately prior, and that the Teller's utterance will have a similar meaning when copied verbatim into a new conversation. Rule-based Nearest-Neighbor Drawer. This Drawer model is the complement to the rule-based nearest-neighbor Teller. It likewise follows a hard-coded rule that the response to each Teller utterance should be the addition of a single clip art to the scene, and makes use of the same database of (Teller utterance, clip art object) tuples collected from the training data. Each Teller utterance the agent receives is compared with the stored tuples using character-level string edit distance. The clip art object from the most similar tuple is selected and added to the canvas by the Drawer. In this section, we describe a neural network approach to the Drawer. Contextual reasoning is an important part of the CoDraw task: each message from the Teller can relate back to what the Drawer has previously heard or drawn, and the clip art pieces it places on the canvas must form a semantically coherent scene. To capture these effects, our model should condition on the past history of the conversation and use an action representation that is conducive to generating coherent scenes. conditions on the current state of the canvas and a BiLSTM encoding of the previous utterance to decide which clip art pieces to add to a scene. The Teller (right) uses an LSTM language model with attention to the scene (in blue) taking place before and after the LSTM. The "thought bubbles" represent intermediate supervision using an auxiliary task of predicting which clip art have not been described yet. In reinforcement learning, the intermediate scenes produced by the drawer are used to calculate rewards. Note that the language used here was constructed for illustrative purposes, and that the messages in our dataset are more detailed and precise. When considering past history, we make the Markovian assumption that the current state of the Drawer's canvas captures all information from the previous rounds of dialog. Thus, the Drawer need only consider the most recent utterance from the Teller and the current canvas to decide what to draw next. We experimented with incorporating additional context -such as previous messages from the teller or the action sequence by which the Drawer arrived at its current canvas configuration -but did not observe any gains in performance. The current state of the canvas is represented using a collection of indicator features and real-valued features. For each of the n c = 58 clip art types, there is an indicator feature for its presence on the canvas, and an indicator feature for each discrete assignment of an attribute to the clip art (e.g. 1 size=small, 1 size=medium, etc.) for a total of n b = 41 binary features. There are additionally two real-valued features that encode the x and y position of the clip art on the canvas, normalized to the 0-1 range. The ing canvas representation is a feature vector v canvas of size n c × (n b + 2), where all features for absent clip art types are set to zero. We run a bi-directional LSTM over the Teller's most recent message and extract the final hidden states for both directions, which we concatenate to form a vector v message. The Drawer is then a feedforward neural network that takes as input v canvas and v message and produces an output vector v action. The action representation v action also consists of n c × (n b + 2) elements and can be thought of as a continuous relaxation of the mostly-discrete canvas encoding. For each clip art type, there is a realvalued score that determines whether a clip art piece of that type should be added to the canvas: a positive score indicates that it should be added as part of the action. During training, a binary crossentropy loss compares these scores with the actions taken by human drawers. v action also contains unnormalized log-probabilities for each attribute-value assignment (e.g. z size=small, z size=medium, etc. for each clip art type); when a clip art piece is added to the canvas, its attributes are assigned to their most-probable values. The log-probabilities are trained using softmax losses. Finally, v action contains two entries for each clip art type that determine the clip art's x, y position if added to the canvas; these elements are trained using an L 2 loss. For our neural Teller models, we adopt an architecture that we call scene2seq. This architecture is a conditional language model over the Teller's side of the conversation with special next-utterance tokens to indicate when the Teller ends its current utterance and waits for a reply from the Drawer. scene is incorporated both before and after each LSTM cell through the use of an attention mechanism. Attention occurs over individual clip art pieces: each clip art object in the ground-truth scene is represented using a vector that is the sum of learned embeddings for different clip art attributes (e.g. e type=Mike, e size=small, etc.) At test time, the Teller's messages are constructed by decoding from the language model using greedy word selection. To communicate effectively, the Teller must keep track of which parts of the scene it has and has not described, and also generate language that is likely to accomplish the task objective when interpreted by the Drawer. We found that training the scene2seq model using a maximum likelihood objective did not in long-term coherent dialogs for novel scenes. Rather than introducing a new architecture to address these deficiencies, we explore reducing them by using alternative training objectives. To better ensure that the model keeps track of which pieces of information it has already communicated, we take advantage of the availability of drawings at each round of the recorded human dialogs and introduce an auxiliary loss based on predicting these drawings. To select language that is more likely to lead to successful task completion, we further fine-tune our Teller models to directly optimize the end-task goal using reinforcement learning. We incorporate state tracking into the scene2seq architecture through the use of an auxiliary loss. This formulation maintains the end-to-end training procedure and keeps test-time decoding exactly the same. The only change is that, at each utterance separator token, the output from the LSTM is used to predict which clip art still need to be described. More precisely, the network must classify whether each clip art in the ground truth has been drawn already or not. The supervisory signal makes use of the fact that the CoDraw dataset records human drawer actions at each round of the conversation, not just at the end. The network outputs a score for each clip art ID, which is connected to a softmax loss for the clip art in the ground truth scene (the scores for absent clip arts do not contribute to the auxiliary loss). We find that adding such a supervisory signal reduces the Teller's propensity for repeating itself or omitting objects. The auxiliary loss helps the agent be more coherent throughout the dialog, but it is still trained to imitate human behavior rather than to complete the downstream task. By training the agents using reinforcement learning (RL), they can learn to use language that is more effective at achieving highfidelity scene reconstructions. In this work we only train the Teller with RL, because the Teller has challenges maintaining a long-term strategy throughout a long dialog, whereas preliminary showed that making local decisions is less detrimental for Drawers. The scene2seq Teller architecture remains unchanged, and each action from the agent is to output a word or one of two special tokens: a next-utterance token and a stop token. After each next-utterance token, our neural Drawer model is used to take an action in the scene and the ing change in scene similarity metric is used as a reward. However, this reward scheme alone has an issue: once all objects in the scene are described, any further messages will not in a change in the scene and have a reward of zero. As a , there is no incentive to end the conversation. We address this by applying a penalty of 0.3 to the reward whenever the Drawer makes no changes to the scene. We train our model with REINFORCE . To evaluate our models, we pair our models with other models, as well as with a human. Human-Machine Pairs. We modified the interface used for data collection to allow human-machine pairs to complete the tasks. Each model plays one game with a human per scene in the test set, and we compare the scene reconstruction quality between different models and with human-human pairs. Script-based Drawer Evaluation. In addition to human evaluation, we would like to have automated evaluation protocols that can quickly estimate the quality of different models. Drawer models can be evaluated by pairing them with a Teller that replays recorded human conversation from a script (a recorded dialog from the dataset) and measuring scene similarity at the end of the dia- Figure 3: A rule-based nearest-neighbor Teller/Drawer pair "trained" on the same data outperforms humans for this scene according to the similarity metric, but the language used by the models doesn't always correspond in meaning to the actions taken. The three panels on the left show a scene from the test set and corresponding human/model reconstructions. The two panels on the right show the Teller message and Drawer action from two rounds of conversation by the machine agents.log. While this setup does not capture the full interactive nature of the task, the Drawer model still receives human descriptions of the scene and should be able to reconstruct it. Our modeling assumptions include not giving Drawer models the ability to ask clarifying questions, which further suggests that script-based evaluation can reasonably measure model quality. Machine-Machine Evaluation. Unlike Drawer models, Teller models cannot be evaluated using a "script" from the dataset. We instead consider an evaluation where a Teller model and Drawer model are paired, and their joint performance is evaluated using the scene similarity metric. Automatically evaluating agents, especially in the machine-machine paired setting, requires some care because a pair of agents can achieve a perfect score while communicating in a shared code that bears no resemblance to natural language. There are several ways such co-adaptation can develop. One is by overfitting to the training data to the extent that it's used as a codebook -we see this with the rule-based nearest-neighbor agents described in Section 4.1, where a Drawer-Teller pair "trained" on the same data outperforms humans on the CoDraw task. An examination of the language, however, reveals that only limited generalization has taken place (see Figure 3). Another way that agents can co-adapt is if they are trained jointly, for example using reinforcement learning. To limit these sources of co-adaptation, we propose a training protocol we call "crosstalk." In this setting, the training data is split in half, and the Teller and Drawer are trained separately on disjoint halves of the training data. When multiple agents are required during training (as with reinforcement learning), the joint training process is run separately for both halves of the training data, but evaluation pairs a Teller from the first partition with a Drawer from the second. This ensures to some extent that the models can succeed only if they have learned generalizable language, and not via a highly specialized codebook specific to model instances. Taking the crosstalk training protocol into account, the dataset split we use for all experiments is: 40% Teller training data (3,994 scenes/dialogs), 40% Drawer training data, 10% development data and 10% testing data. Results for our models are shown in TAB1. All numbers are scene similarities, averaged across scenes in the test set. Neural Drawer Performs the Best. In the script setting, our neural Drawer is able to outperform the rule-based nearest-neighbor baseline (3.39 vs. 0.94) and close most of the gap between baseline (0.94) and human performance (4.17).Validity of Script-Based Drawer Evaluation. To test the validity of script-based Drawer evaluation -where a Drawer is paired with a Teller that recites the human script from the dataset corresponding to the test scenes -we include from interactively pairing human Drawers with a Teller that recites the scripted messages. While average scene similarity is lower than when using live human Tellers (3.83 vs. 4.17), the scripts are sufficient to achieve over 91% of the effectiveness of the Benefits of Intermediate Supervision and Goal-Driven Training. Pairing our models with humans shows that the scene2seq Teller model trained with imitation learning is worse than the rulebased nearest-neighbor baseline (2.69 vs. 3.21), but that the addition of an auxiliary loss followed by fine-tuning with reinforcement learning allow it to outperform the baseline (3.65 vs. 3.21). However, there is still a gap between to human Tellers (3.65 vs. 4.17). Many participants in our human study noted that they received unclear instructions from the models they were paired with, or expressed frustration that their partners could not answer clarifying questions as a way of resolving such situations. Recall that our Teller models currently ignore any utterances from the Drawer. Correlation Between Fully-automated and Human-machine Evaluation. We also report the of paired evaluation for different Teller models and our best Drawer, showing that the relative rankings of the different Teller types match those we see when models are paired with humans. This shows that automated evaluation while following the crosstalk training protocol is a suitable automated proxy for human-evaluation. The errors made by Teller reflect two key challenges posed by the CoDraw task: reasoning about the context of the conversation and drawing, and planning ahead to fully and effectively communicate the information required. A common mistake the rule-based nearest-neighbor Teller makes is to reference objects that are not present in the current scene. Figure 3 shows an example (second panel from the right) where the Teller has copied a message referencing a "swing" that does not exist in the current scene. In a sample of 5 scenes from the test set, the rule-based nearest-neighbor describes a non-existent object 11 times, compared to just 1 time for the scene2seq Teller trained with imitation learning. The scene2seq Teller, on the other hand, frequently describes clip art pieces multiple times or forgets to mention some of them: in the same sample of scenes, it re-describes an object 10 times (vs. 2 for the baseline) and fails to mention 11 objects (vs. 2.) The addition of an auxiliary loss and RL fine-tuning reduces these classes of errors while avoiding frequent descriptions of irrelevant objects (0 references to non-existent objects, 3 instances of re-describing an object, and 4 objects omitted.)On the Drawer side, the most salient class of mistakes made by the neural network model is semantically inconsistent placement of multiple clip art pieces. Several instances of this can be seen in FIG0 in the Appendix D, where the Drawer places a hat in the air instead of on a person's head, or where the drawn clip art pieces overlap in a visually unnatural way. Qualitative examples of both human and model behavior are provided in Appendix D. In this paper, we introduce CoDraw: a collaborative task designed to facilitate learning of effective natural language communication in a grounded context. The task combines language, perception, and actions while permitting automated goal-driven evaluation both at the end and as a measure of intermediate progress. We introduce a dataset and models for this task, and propose a crosstalk training + evaluation protocol that is more generally applicable to studying emergent communication. The models we present in this paper show levels of task performance that are still far from what humans can achieve. Long-term planning and contextual reasoning as two key challenges for this task that our models only begin to address. We hope that the grounded, goal-driven communication setting that CoDraw is a testbed for can lead to future progress in building agents that can speak more naturally and better maintain coherency over a long dialog, while being grounded in perception and actions. A.1 INTERFACE Figure 4 shows the interface for the Teller, and Figure 5 shows the interface for the Drawer. Following previous works, Drawers are given 20 clip art objects selected randomly from the 58 clip art objects in the library, while ensuring that all objects required to reconstruct the scene are available. Your fellow Turker will ask you questions about your secret scene. 1Your objective is to help the fellow Turker recreate the scene. You typically describe the details of the image and/or answer their questions. Use Chance Finish HIT! Figure 5: User interface for a Drawer. The Drawer has an empty canvas and a randomly generated drawing palette of Mike, Jenny, and 18 other objects, chosen from a library of 58 clip arts. We ensure that using the available objects, Drawer can fully reproduce the scene. Using the library, the Drawer can draw on the canvas in a drag-and-drop fashion. Drawer can also send messages using a given input box. However, the peek button is disabled. Only the Teller can use it. We found that approximately 13.6% of human participants disconnect voluntarily in an early stage of the session. We paid participants who stayed in the conversation and had posted at least three messages. However, we exclude those incomplete sessions in the dataset, and only use the completed sessions. There are 616 unique participants represented in our collected data. Among these workers, the 5 most active have done 26.63% of all finished tasks (1,419, 1,358, 1,112, 1,110, and 1,068 tasks). Across all workers, the maximum, median, and minimum numbers of tasks finished by a worker are 1,419, 3, and 1, respectively.*Collected 9,993 sessions as of Apr 19 2017 The number of sessions.0K.4K.8K1.2K The number of sessions.0K.4K.8K1.2K The CoDraw dataset consists of 9,993 dialogs consisting of a total of 138K utterances. Each dialog describes a distinct abstract scene. Messages. FIG3 shows the distribution of message lengths for both Drawers and Tellers. Drawer messages tend to be short (the median length is 1 accounts for 62% of messages), but there does exist a heavy tail where the Drawer asks clarifying questions about the scene. Teller message length have a more smooth distribution with a median length of 16 tokens. The size of vocabulary is 4,555: since conversations describe abstract scenes consisting of a limited number of clip art types, the vocabulary is relatively small compared to tasks involving real images. Rounds. FIG3 shows the distribution of the numbers of conversational rounds for dialog sessions. Most interactions are shorter than 20 rounds, median being 7.Durations. In FIG3 we see that the median session duration is 6 minutes. We had placed a 20-minute maximum limit on each session. Scores. FIG4 shows the distribution of scene similarity scores throughout the dataset. FIG5 shows the progress of scene similarity scores over the rounds of a conversation. An average conversations is done improving the scene similarity after about 5 rounds, but for longer conversations that continue to 23 rounds, there is still room for improvement. Given a ground-truth scene C and a predicted sceneĈ (where the presence of a clip art type c in the scene C is indicated by c ∈ C) scene similarity s is defined as: DISPLAYFORM0 where f (c) =w 0 − w 1 1 clip art piece c faces the wrong direction − w 2 1 clip art piece c is Mike or Jenny and has the wrong facial expression − w 3 1 clip art piece c is Mike or Jenny and has the wrong body pose − w 4 1 clip art piece c has the wrong size DISPLAYFORM1 Here x c and y c refer to the position of the clip art piece in the ground-truth scene,x c andŷ c refer to its position in the predicted scene, and W, H are the width and height of the canvas, respectively. We use parameters w = [5, 1, 0.5, 0.5, 1, 1, 1, 1], which provides a balance between the different components and ensures that scene similarities are constrained to be between 0 and 5.D QUALITATIVE EXAMPLES Figure 9 shows some examples of scenes and dialogs from the CoDraw dataset. The behavior of our Drawer and Teller models on a few randomly-selected scenes is illustrated in FIG0, and 12. Figure 9: Examples from the Collaborative Drawing (CoDraw) dataset, chosen at random from the test set. The images depict the Drawer's canvas after each round of conversation. From left to right, we show rounds one through four, then the last round, followed by the ground truth scene. The corresponding conversations between the Teller (T) and Drawer (D) are shown below the images. Note that there is no restriction on which of the two participants begins or ends the dialog. small hot air balloon top right B2 in front of tree is boy, he is to the left part of tree and is covering the curve up. he is angry, standing, arms, out facing left small girl, running, facing right, surprised, 1 " from bottom, 1 2 " from left small hot balloon on right corner, half " from top.large bear on left faced right B3 the head of surprised girl is on front the trunk. she is like running and faces right.small pine tree behind her, bottom of trunk at horizon, bottom of trunk at horizon, small boy in front of tree, head touching bottom of tree, standing, smiling, facing right, holding a hot dog in left hand on center, a mad mike with hands front facing left. FIG0: A comparison of the descriptions generated by each of our Teller models for two randomly-sampled scenes from the test set.

Represent the TLDR for finding the corresponding scientific article:

We introduce a dataset, models, and training + evaluation protocols for a collaborative drawing task that allows studying goal-driven and perceptually + actionably grounded language generation and understanding.

represent the short summary for finding the corresponding scientific article (long)

We propose a novel Intrinsically Motivated Goal Exploration architecture with unsupervised learning of goal space representations, and evaluate how various implementations enable the discovery of a diversity of policies.