Question: Represent the scientific article to find a short TLDR

Question: Generative modeling of high dimensional data like images is a notoriously difficult and ill-defined problem. In particular, how to evaluate a learned generative model is unclear. In this paper, we argue that *adversarial learning*, pioneered with generative adversarial networks (GANs), provides an interesting framework to implicitly define more meaningful task losses for unsupervised tasks, such as for generating "visually realistic" images. By relating GANs and structured prediction under the framework of statistical decision theory, we put into light links between recent advances in structured prediction theory and the choice of the divergence in GANs. We argue that the insights about the notions of "hard" and "easy" to learn losses can be analogously extended to adversarial divergences. We also discuss the attractive properties of parametric adversarial divergences for generative modeling, and perform experiments to show the importance of choosing a divergence that reflects the final task. For structured prediction and data generation the notion of final task is at the same time crucial and not well defined. Consider machine translation; the goal is to predict a good translation, but even humans might disagree on the correct translation of a sentence. Moreover, even if we settle on a ground truth, it is hard to define what it means for a candidate translation to be close to the ground truth. In the same way, for data generation, the task of generating pretty pictures or more generally realistic samples is not well defined. Nevertheless, both for structured prediction and data generation, we can try to define criteria which characterize good solutions such as grammatical correctness for translation or non-blurry pictures for image generation. By incorporating enough criteria into a task loss, one can hope to approximate the final task, which is otherwise hard to formalize. Supervised learning and structured prediction are well-defined problems once they are formulated as the minimization of such a task loss. The usual task loss in object classification is the generalization error associated with the classification error, or 0-1 loss. In machine translation, where the goal is to predict a sentence, a structured loss, such as the BLEU score BID37, formally specifies how close the predicted sentence is from the ground truth. The generalization error is defined through this structured loss. In both cases, models can be objectively compared and evaluated with respect to the task loss (i.e., generalization error). On the other hand, we will show that it is not as obvious in generative modeling to define a task loss that correlates well with the final task of generating realistic samples. Traditionally in statistics, distribution learning is formulated as density estimation where the task loss is the expected negative-log-likelihood. Although log-likelihood works fine in low-dimension, it was shown to have many problems in high-dimension. Among others, because the Kullback-Leibler is too strong of a divergence, it can easily saturate whenever the distributions are too far apart, which makes it hard to optimize. Additionally, it was shown in BID47 that the KL-divergence is a bad proxy for the visual quality of samples. In this work we give insights on how adversarial divergences BID26 can be considered as task losses and how they address some problems of the KL by indirectly incorporating hard-to-define criteria. We define parametric adversarial divergences as the following: DISPLAYFORM0 where {f φ : X → R d ; φ ∈ Φ} is a class of parametrized functions, such as neural networks, called the discriminators in the Generative Adversarial Network (GAN) framework BID15. The constraints Φ and the function ∆: R d × R d → R determine properties of the ing divergence. Using these notations, we adopt the view 1 that training a GAN can be seen as training a generator network q θ (parametrized by θ) to minimize the parametric adversarial divergence Div NN (p||q θ), where the generator network defines the probability distribution q θ over x. Our contributions are the following:• We show that compared to traditional divergences, parametric adversarial divergences offer a good compromise in terms of sample complexity, computation, ability to integrate prior knowledge, flexibility and ease of optimization.• We relate structured prediction and generative adversarial networks using statistical decision theory, and argue that they both can be viewed as formalizing a final task into the minimization of a statistical task loss.• We explain why it is necessary to choose a divergence that adequately reflects our final task in generative modeling. We make a parallel with in structured learning (also dealing with high-dimensional data), which quantify the importance of choosing a good objective in a specific setting.• We explore with some simple experiments how the properties of the discriminator transfer to the adversarial divergence. Our experiments suggest that parametric adversarial divergences are especially adapted to problems such as image generation, where it is hard to formally define a perceptual loss that correlates well with human judgment.• We illustrate the importance of having a parametric discriminator by running experiments with the true (nonparametric) Wasserstein, and showing its shortcomings on complex datasets, on which GANs are known to perform well.• We perform qualitative and quantitative experiments to compare maximum-likelihood and parametric adversarial divergences under two settings: very high-dimensional images, and learning data with specific constraints. Here we briefly introduce the structured prediction framework because it can be related to generative modeling in some ways. We will later link them formally, and present insights from recent theoretical to choose a better divergence. We also unify parametric adversarial divergences with traditional divergences in order to compare them in the next section. The goal of structured prediction is to learn a classifier h θ: X → Y which predicts a structured output y from an input x. The key difficulty is that Y usually has size exponential in the input 2 (e.g. it could be all possible sequence of symbols with a given length). Being able to handle this exponentially large set of outputs is one of the key challenges in structured prediction because it makes traditional multi-class classification methods unusable in general.3 Standard practice in structured prediction BID46 BID9 BID38 is to consider predictors based on score functions h θ (x) = arg max y ∈Y s θ (x, y), where s θ: X × Y → R, called the score/energy function BID23, assigns a score to each possible label y for an input x. Typically, 1 We focus in this paper on the divergence minimization perspective of GANs. There are other views, such as those based on game theory BID2, ratio matching and moment matching BID29.2 Additionally, Y might depend on the input x, but we ignore this effect for clarity of exposition. 3 Such as ones based on maximum likelihood.as in structured SVMs BID46, the score function is linear: s θ (x, y) = θ, g(x, y), where g(·) is a predefined feature map. Alternatively, the score function could also be a learned neural network BID5.In order to evaluate the predictions objectively, we need to define a task-dependent structured loss (y, y ; x) which expresses the cost of predicting y for x when the ground truth is y. We discuss the relation between the loss function and the actual final task in Section 4.2. The goal is then to find a parameter θ which minimizes the generalization error: DISPLAYFORM0 Directly minimizing is often an intractable problem; this is the case when the structured loss is the 0-1 loss BID1. Instead, the usual practice is to minimize a surrogate loss et al., 2006) which has nicer properties, such as subdifferentiability or convexity, to get a tractable optimization problem. The surrogate loss is said to be consistent when its minimizer is also a minimizer of the task loss. DISPLAYFORM1 A simple example of structured prediction task is machine translation. Suppose we want to translate French sentences to English; the input x is then a sequence of French words, and the output y is a sequence of English words belonging to a dictionary D with typically |D| ≈ 10000 words. If we restrict the output sequence to be shorter than T words, then |Y| = |D| T, which is exponential. An example of desirable criterion is to have a translation with many words in common with the ground truth, which is typically enforced using BLEU scores to define the task loss. Because we will compare properties of adversarial and traditional divergences throughout this paper, we choose to first unify them with a formalism similar to BID45; BID26: DISPLAYFORM0 Under this framework we give some examples of traditional nonparametric divergences:• ψ-divergences with generator function ψ (which we call f-divergences) can be written in dual form BID33 4 Div ψ (p||q θ) = sup DISPLAYFORM1 where ψ * is the convex conjugate. Depending on ψ, one can obtain any ψ-divergence such as the (reverse) Kullback-Leibler, the Jensen-Shannon, the Total Variation, the ChiSquared 5.• Wasserstein-1 distance induced by an arbitrary norm · and its corresponding dual norm · * BID45: DISPLAYFORM2 which can be interpreted as the cost to transport all probability mass of p into q, where x − x is the unit cost of transporting x to x.• Maximum Mean Discrepancy: DISPLAYFORM3 where (H, K) is a Reproducing Kernel Hilbert Space induced by a Kernel K(x, x) on X with the associated norm · H. The MMD has many interpretations in terms of momentmatching BID24. 4 The standard form is Ex∼q θ [ψ( DISPLAYFORM4 . Some ψ require additional constraints, such as ||f ||∞ ≤ 1 for the Total Variation. Table 1 : Properties of Divergences. Explicit and Implicit models refer to whether the density q θ (x) can be computed. p is the number of parameters of the parametric discriminator. Sample complexity and computational cost are defined and discussed in Section 3.1, while the ability to integrate desirable properties of the final loss is discussed in Section 3.2. Although f-divergences can be estimated with Monte-Carlo for explicit models, they cannot be easily computed for implicit models without additional assumptions (see text). Additionally, by design, they cannot integrate a final loss directly. The nonparametric Wasserstein can be computed iteratively with the Sinkhorn algorithm, and can integrate the final loss in its base distance, but requires exponentially many samples to estimate. Maximum Mean Discrepancy has good sample complexity, can be estimated analytically, and can integrate the final loss in its base distance, but it is known to lack discriminative power for generic kernels, as discussed below. Parametric adversarial divergences have reasonable sample complexities, can be computed iteratively with SGD, and can integrate the final loss in the choice of class of discriminators. DISPLAYFORM5 In particular, the parametric Wasserstein has the additional possibility of integrating the final loss into the base distance. In the optimization problems FORMULA4 and FORMULA5, whenever f is additionally constrained to be in a given parametric family, the associated divergence will be termed a parametric adversarial divergence. In practice, that family will typically be specified as a neural network architecture, so in this work we will use the term neural adversarial divergences interchangeably with the slightly more generic parametric adversarial divergence. For instance, the parametric adversarial Jensen-Shannon optimized in GANs corresponds to with specific ψ BID33, while the parametric adversarial Wasserstein optimized in WGANs corresponds to where f is a neural network. See BID26 for interpretations and a review and interpretation of other divergences like the Wasserstein with entropic smoothing BID3, energy-based distances BID24 which can be seen as adversarial MMD, and the WGAN-GP BID18 objective. We argue that parametric adversarial divergences have many good properties which make them attractive for generative modeling. In this section, we compare them to traditional divergences in terms of sample complexity and computational cost (Section 3.1), and ability to integrate criteria related to the final task (Section 3.2). We also discuss the shortcomings of combining the KL-divergence with generators that have a special structure in Section 3.3. We refer the reader to the Appendix for additional interesting properties of parametric adversarial divergences: the optimization and stability issues are discussed in Appendix A.1, the fact that parametric adversarial divergences only make the assumption that one can sample from the generative model, and provide useful learning signal even when their nonparametric counterparts are not well-defined, is discussed in Appendix A.2. Since we want to learn from finite data, we would like to know how well empirical estimates of a divergence approximate the population divergence. In other words, we want to control the sample complexity, that is, how many samples n do we need to have with high probability that |Div(p||q) − Div(p n || q n)| ≤, where > 0, and p n, q n are empirical distributions associated with p, q. Sample complexities for adversarial and traditional divergences are summarized in Table 1.For explicit models which allow evaluating the density q θ (x), one could use Monte-Carlo to evaluate the f-divergence with sample complexity n = O(1/ 2), according to the Central-Limit theorem. For implicit models, there is no one good way of estimating f-divergences from samples. There are some techniques for it BID32 BID30 BID43, but they all make additional assumptions about the underlying densities (such as smoothness), or they solve the dual in a restricted family, such as a RKHS, which makes the divergences no longer f-divergences. Parametric adversarial divergences can be formulated as a classification/regression problem with a loss depending on the specific adversarial divergence. Therefore, they have a reasonable sample complexity of O(p/ 2), where p is the VC-dimension/number of parameters of the discriminator BID2, and can be solved using classic stochastic gradient methods. A straightforward nonparametric estimator of the Wasserstein is simply the Wasserstein distance between the empirical distributions p n and q n, for which smoothed versions can be computed in O(n 2) using specialized algorithms such as Sinkhorn's algorithm BID11 or iterative Bregman projections BID7. However, this empirical Wasserstein estimator has sample complexity n = O(1/ d+1) which is exponential in the number of dimensions (see , Corollary 3.5). Thus the empirical Wasserstein is not a viable estimator in high-dimensions. Maximum Mean Discrepancy admits an estimator with sample complexity n = O(1/ 2), which can be computed analytically in O(n 2). More details are given in the original MMD paper BID16. One should note that MMD depends fundamentally on the choice of kernel. As the sample complexity is independent of the dimension of the data, one might believe that the MMD estimator behaves well in high dimensions. However, it was experimentally illustrated in Dziugaite et al. FORMULA0 that with generic kernels like RBF, MMD performs poorly for MNIST and Toronto face datasets, as the generated images have many artifacts and are clearly distinguishable from the training dataset. See Section 3.2 for more details on the choice of kernel. It was also shown theoretically in BID40 that the power of the MMD statistical test can drop polynomially with increasing dimension, which means that with generic kernels, MMD might be unable to discriminate well between high-dimensional generated and training distributions. Note that comparing divergences in terms of sample complexity can give good insights on what is a good divergence, but should be taken with a grain of salt as well. On the one hand, the sample complexities we give are upper-bounds, which means the estimators could potentially converge faster. On the other hand, one might not need a very good estimator of the divergence in order to learn in some cases. This is illustrated in our experiments with the empirical Wasserstein (Section 6) which has bad sample complexity but yields reasonable . In Section 4, we will argue that in structured prediction, optimizing for the right task losses is more meaningful and can make learning considerably easier. Similarly in generative modeling, we would like divergences to integrate criteria that characterize the final task. We discuss that although not all divergences can easily integrate final task-related criteria, adversarial divergences provide a way to do so. Pure f-divergences cannot directly integrate any notion of final task, 6 at least not without tweaking the generator. The Wasserstein distance and MMD are respectively induced by a base metric d(x, x) and a kernel K(x, x). The metric and kernel give us the opportunity to specify a task by letting us express a (subjective) notion of similarity. However, the metric and kernel generally have to be defined by hand, as there is no obvious way to learn them end-to-end. For instance, BID14 learn to generate MNIST by minimizing a smooth Wasserstein based on the L2-distance, while Dziugaite et al. FORMULA0; BID25 also learn to generate MNIST by minimizing the MMD induced by kernels obtained externally: either generic kernels based on the L2-distance or on autoencoder features. However, the seems to be limited to simple datasets. Recently there has been a surge of interest in combining MMD with kernel learning, with convincing on LSUN, CelebA and ImageNet images. BID31 learn a feature map and try to match its mean and covariance, BID24 learn kernels end-to-end, while BID6 do end-to-end learning of energy distances, which are closely related to MMD.Parametric adversarial divergences are defined with respect to a parametrized class of discriminators, thus changing properties of the discriminator is a primary way to affect the associated divergence. The form of the discriminator may determine what aspects the divergence will be sensitive or blind to. For instance using a convolutional network as the discriminator may render the divergence insen-sitive to small image translations. Additionally, the parametric adversarial Wasserstein distance can also incorporate a custom metric. In Section 6 we give interpretations and experiments to assess the relation between the discriminator and the divergence. In some cases, imposing a certain structure on the generator (e.g. a Gaussian or Laplacian observation model) yields a Kullback-Leibler divergence which involves some form of component-wise distance between samples, reminiscent of the Hamming loss (see Section 4.3) used in structured prediction. However, doing maximum likelihood on generators having an imposed special structure can have drawbacks which we detail here. For instance, the generative model of a typical variational autoencoder can be seen as an infinite mixture of Gaussians BID19. The loglikelihood thus involves a "reconstruction loss", a pixel-wise L2 distance between images analogous to the Hamming loss, which makes the training relatively easy and very stable. However, the Gaussian is partly responsible for the VAE's inability to learn sharp distributions. Indeed it is a known problem that VAEs produce blurry samples, in fact even if the approximate posterior matches exactly the true posterior, which would correspond to the evidence lower-bound being tight, the output of the VAE would still be blurry. Other examples are autoregressive models such as recurrent neural networks BID28 which factorize naturally as log q θ (x) = i log q θ (x i |x 1, .., x i−1), and PixelCNNs BID34. Training autoregressive models using maximum likelihood in teacher-forcing BID21: each ground-truth symbol is fed to the RNN, which then has to maximize the likelihood of the next symbol. Since teacher-forcing induces a lot of supervision, it is possible to learn using maximumlikelihood. Once again, there are similarities with the Hamming loss because each predicted symbol is compared with its associated ground truth symbol. However, among other problems, there is a discrepancy between training and generation. Sampling from q θ would require iteratively sampling each symbol and feeding it back to the RNN, giving the potential to accumulate errors, which is not something that is accounted for during training. See BID22 and references therein for more principled approaches to sequence prediction with autoregressive models. In this section, we try to provide insights in order to design the best adversarial divergence for our final task. After establishing the relationship between structured prediction and generative adversarial networks, we review theoretical on the choice of objectives in structured prediction, and discuss their interpretation in generative modeling. We frame the relationship of structured prediction and GANs using the framework of statistical decision theory. Assume that we are in a world with a set P of possible states and that we have a set A of actions. When the world is in the state p ∈ P, the cost of playing action a ∈ A is the (statistical) task loss L p (a). The goal is to play the action minimizing the task loss. Generative models with Maximum Likelihood. The set P of possible states is the set of available distributions {p} for the data x. The set of actions A is the set of possible distributions{q θ ; θ ∈ Θ} for the model and the task loss is the negative log-likelihood, DISPLAYFORM0 Structured prediction. The set P of possible states is the set of available distribution {p} for (x, y). The set of actions A is the set of prediction functions {h θ ; θ ∈ Θ} and the task loss is the generalization error: DISPLAYFORM1 where: Y × Y × X → R is a structured loss function. the minimization of a statistical task loss. One starts from a useful but illdefined final task, and devises criteria that characterize good solutions. Such criteria are integrated into the statistical task loss, which is the generalization error in structured prediction, and the adversarial divergence in the GAN framework. The hope is that minimizing the statistical task loss effectively solves the final task. GANs. The set P of possible states is the set of available distributions {p} for the data x. The set of actions A is the set of distributions {q θ ; θ ∈ Θ} that the generator can learn, and the task loss is the adversarial divergence DISPLAYFORM2 Under this unified framework, the prediction function h θ is analogous to the generative model q θ, while the choice of the right structured loss can be related to ∆ and to the choice of the discriminator family F which will induce a good adversarial divergence. We will further develop this analogy in Section 4.2. As discussed in the introduction, structured prediction and data generation involve a notion of final task which is at the same time crucial and not well defined. Nevertheless, for both we can try to define criteria which characterize good solutions. We would like the statistical task loss (introduced in Section 4.1), which corresponds to the generalization error in structured prediction, and the adversarial divergence in generative modeling, to incorporate task-related criteria. One way to do that is to choose a structured loss that reflects the criteria of interest, or analogously to choose a class of discriminators, like a CNN architecture, such that the ing adversarial divergence has good invariance properties. The whole process of building statistical task losses adapted to a final task, using the right structured losses or discriminators, is represented in FIG0.For many prediction problems, the structured prediction community has engineered structured loss functions which induce properties of interest on the learned predictors. In machine translation, a commonly considered property of interest is for candidate translations to contain many words in common with the ground-truth; this has given rise to the BLEU score which counts the percentage of candidate words appearing in the ground truth. In the context of image segmentation, BID35 have compared various structured loss functions which induces different properties on the predicted mask. In the same vein as structured loss functions, adversarial divergences can be built to induce certain properties on the generated data. We are more concerned with generating realistic samples than having samples which are very similar with the training set; we actually want to extrapolate some properties of the true distribution from the training set. For instance, in the DCGAN, the discriminator has a convolutional architecture, which makes it potentially robust to small deformations that would not affect the visual quality of the samples significantly, while still making it able to detect blurry samples, which is aligned with our objective of generating realistic samples. Intuition on the Flexibility of Losses. In this section we get insights from the convergence of in structured prediction. They show in a specific setting that some "weaker" structured loss functions are easier to learn than some stronger loss functions. In some sense, their formalize the intuition in generative modeling that learning with "weaker" divergences is easier ) and more intuitive BID26 than stronger divergences. In structured prediction, strong losses such as the 0-1 loss are hard to learn with because they do not give any flexibility on the prediction; the 0-1 loss only tells us whether a prediction is correct or not, and consequently does not give any clue about how close the prediction is to the ground truth. To get enough learning signal, we roughly need as many training examples as the number of possible outputs |Y|, which is exponential in the dimension of y and thus inefficient. Conversely, weaker losses like the Hamming loss have more flexibility; because they tell us how close a prediction is to the ground truth, less examples are needed to generalize well. The theoretical proved by formalize that intuition in a specific setting. Theory to Back the Intuition. In a non-parametric setting (details and limitations in Appendix B), formalize the intuition that weaker structured loss functions are easier to optimize. Specifically, they compare the 0-1 loss 0−1 (y, y) =1 {y = y} to the Hamming lossHam (y, y) = 1 T T t=1 1{y t = y t}, when y decomposes as T = log 2 |Y| binary variables (y t) 1≤t≤T. They derive a worst case sample complexity needed to obtain a fixed error > 0. For the 0-1 loss, they obtain a sample complexity of O(|Y|/ 2) which is exponential in the dimension of y. However, for the Hamming loss, under certain constraints (see, section on exact calibration functions) they obtain a much better sample complexity of O(log 2 |Y|/ 2) which is polynomial in the number of dimensions, whenever certain constraints are imposed on the score function. Thus their suggest that choosing the right structured loss, like the weaker Hamming loss, might make training exponentially faster. Insights and Relation with Adversarial Divergences.'s theoretical confirm our intuition that weaker losses are easier to optimize, and quantify in a specific setting how much harder it is to learn with strong structured loss functions, like the 0-1 loss, than with weaker ones, like the Hamming loss (here, exponentially harder). Under the framework of statistical decision theory (introduced Section 4.1), their can be related to analogous in generative modeling BID26 showing that it can be easier to learn with weaker divergences than with stronger ones. In particular, one of their arguments is that distributions with disjoint support can be compared in weaker topologies like the the one induced by the Wasserstein but not in stronger ones like the the one induced by the Jensen-Shannon. Closest to our work are the following two papers. BID2 argue that analyzing GANs with a nonparametric (optimal discriminator) view does not really make sense, because the usual nonparametric divergences considered have bad sample complexity. They also prove sample complexities for parametric divergences. BID26 prove under some conditions that globally minimizing a neural divergence is equivalent to matching all moments that can be represented within the discriminator family. They unify parametric divergences with nonparametric divergences and introduce the notion of strong and weak divergence. However, both those works do not attempt to study the meaning and practical properties of parametric divergences. In our work, we start by introducing the notion of final task, and then discuss why parametric divergences can be good task losses with respect to usual final tasks. We also perform experiments to determine properties of some parametric divergences, such as invariance, ability to enforce constraints and properties of interest, as well as the difference with their nonparametric counterparts. Finally, we unify structured prediction and generative modeling, which could give a new perspective to the community. The following papers are also related to our work because of one of the following aspects: unifying divergences, analyzing their statistical properties, giving other interpretations of generative modeling, improving GANs, criticizing maximum-likelihood as a objective for generative modeling, and other reasons. Before the first GAN paper, BID45 unify traditional IPMs, analyze their statistical properties, and propose to view them as classification problems. Similarly, BID41 show that computing a divergence can be formulated as a classification problem. Later, BID33 generalize the GAN objective to any adversarial f-divergence. However, the first papers to actually study the effect of restricting the discriminator to be a neural network instead of any function are the MMD-GAN papers: BID25; Dziugaite et al. FORMULA0; BID24; BID31 and BID6 who give an interpretation of their Figure 2: Images generated by the network after training with the Sinkorn-Autodiff algorithm on MNIST dataset (left) and CIFAR-10 dataset (right). One can observe than although the network succeeds in learning MNIST, it is unable to produce convincing and diverse samples on the more complex CIFAR-10. energy distance framework in terms of moment matching. BID29 give many interpretations of generative modeling, including moment-matching, divergence minimization, and density ratio matching. On the other hand, work has been done to better understand the GAN objective in order to improve its stability BID44. Subsequently, introduce the adversarial Wasserstein distance which makes training much more stable, and BID18 improve the objective to make it more practical. Regarding model evaluation, BID47 contains an excellent discussion on the evaluation of generative models, they show in particular that log-likelihood is not a good proxy for the visual quality of samples. compare parametric adversarial divergence and likelihood objectives in the special case of RealNVP, a generator with explicit density, and obtain better visual with the adversarial divergence. Concerning theoretical understanding of learning in structured prediction, some recent papers are devoted to theoretical understanding of structured prediction such as BID10 and BID27 which propose generalization error bounds in the same vein as but with data dependencies. One contribution of the present paper is to have taken these from the prior literature and put them in perspective in an attempt to provide a more principled view of the nature and usefulness of parametric divergences, in comparison to traditional divergences. To the best of our knowledge, we are also the first to make a link between the generalization error of structured prediction and the adversarial divergence in generative modeling. Importance of Sample Complexity. Since the sample complexity of the nonparametric Wasserstein is exponential in the dimension (Section 3.1), we check experimentally whether training a generator to minimize the nonparametric Wasserstein distance fails in high dimensions. We implement the Sinkhorn-AutoDiff algorithm BID14 to compute the entropy-regularized L2-Wasserstein distance between minibatches of training images and generated images. Figure 2 shows generated samples after training with the Sinkhorn-Autodiff algorithm on both MNIST and CIFAR-10 dataset. On MNIST, the network manages to produce decent but blurry images. However, on CIFAR-10, which is a much more complex dataset, the network fails to produce meaningful samples, which would suggest that indeed the nonparametric Wasserstein should not be used for generative modeling when the (effective) dimensionality is high. This is to be contrasted with the recent successes in image generation of the parametric Wasserstein BID18, which also has much better sample complexity than the nonparametric Wasserstein. Robustness to Transformations. Intuitively, small rotations should not significantly affect the realism of images, while additive noise should. We study the robustness of various parametric adversarial divergences to rotations and additive noise by plotting the evolution of the divergence between MNIST and rotated/noisy versions of it, as a function of the amplitude of transformation. We consider three discriminators (linear, 1-layer-dense, 2-layer-cnn) combined with two formulations, parametric Jensen-Shannon (ParametricJS) and parametric Wasserstein (ParametricW). Ideally, good divergences should vary smoothly (be robust) with respect to the amplitude of the transformation. For rotations FIG2 ) and all discriminators except the linear, ParametricJS saturates at its maximal value, even for small values of rotation, whereas the Wasserstein distance varies much more smoothly, which is consistent with the example given by. The fact that the linear ParametricJS does not saturate for rotations shows that the architecture of the discriminator has a significant effect on the induced parametric adversarial divergence, and confirms that there is a conceptual difference between the true JS and ParametricJS, and even among different ParametricJS. For additive Gaussian noise FIG2 ), the linear discriminator is unable to distinguish the two distributions (it only sees the means of the distributions), whereas more complex architectures like CNNs do. In that sense the linear discriminator is too weak for the task, or not strict enough BID26, which suggests that a better divergence involves trading off between robustness and strength. Learning High-dimensional Data. We collect Thin-8, a dataset of about 1500 handwritten images of the digit "8", with a very high resolution of 512 × 512, and augment them with elastic deformations. Because the pen strokes are relatively thin, we expect any pixel-wise distance to be uninformative, because the images are dominated by pixels, and because with high probability, any two "8' will intersect on no more than a little area. We train a convolutional VAE and a WGAN-GP BID18, henceforth simply denoted GAN, using nearly the same architectures (VAE decoder similar to GAN generator, VAE encoder similar to GAN discriminator), with 16 latent variables, on the following resolutions: 32 × 32, 128 × 128 and 512 × 512. Generated samples are shown in FIG3. Indeed, we observe that the VAE, trained to minimize the evidence lower bound on maximum-likelihood, fails to generate convincing samples in high-dimensions: they are blurry, pixel values are gray instead of being white, and some samples look like the average of many digits. On the contrary, the GAN can generate sharp and realistic samples even in 512 × 512. Our hypothesis is that the discriminator learns moments which are easier to match than it is to directly match the training set with maximum likelihood. Since we were able to perfectly generate high-resolution digits, an additional insight of our experiment is that the main difficulty in generating high-dimensional natural images (like ImageNet and LSUN bedrooms) resides not in high resolution itself, but in the intrinsic complexity of the scenes. Such complexity can be hidden in low resolution, which might explain recent successes in generating images in low resolution but not in higher ones. Learning Visual Hyperplanes. We design the visual hyperplane task to be able to compare VAEs and GANs quantitatively rather than simply inspecting the quality of their generated images. We create a new dataset by concatenating sets of 5 images from MNIST, such that those digits sum up to 25. We train a VAE and a WGAN-GP (henceforth simply denoted GAN) on this new dataset (we used 4504 combinations out of the 5631 possible combinations for training). Both model share the same architecture for generator network and use 200 latent variables. With the help of a MNIST classifier, we automatically recognize and sum up the digits in each generated sample. FIG4 shows the distributions of the sums of the digits generated by the VAE and GAN 7. We can see that the GAN distribution is more peaked and centered around the target 25, while the VAE distribution is less precise and not centered around the target. In that respect, the GAN was better than the VAE at capturing the particular aspects and constraints of the data distribution (summing up to 25). One, and 512×512 (right column). Note how the GAN samples are always crips and realistic across all resolutions, while the VAE samples tend to be blurry with gray pixel values in high-resolution. We can also observe some averaging artifacts in the top-right 512x512 VAE sample, which looks like the average of two "8". More samples can be found in Section C.2 of the Appendix. and Independent Baseline (gray). The latter draws digits independently according to their empirical marginal probabilities, which corresponds to fitting independent multinomial distributions over digits using maximum likelihood. WGAN-GP beats largely both VAE and Indepedent Baseline as it gives a sharper distribution centered in the target sum 25. possible explanation is that since training a classifier to recognize digits and sum them up is not hard in a supervised setting, it could also be relatively easy for a discriminator to enforce such a constraint. We gave arguments in favor of using adversarial divergences rather than traditional divergences for generative modeling, the most important of which being the ability to account for the final task. After linking structured prediction and generative modeling under the framework of statistical decision theory, we interpreted recent from structured prediction, and related them to the notions of strong and weak divergences. Moreover, viewing adversarial divergences as statistical task losses led us to believe that some adversarial divergences could be used as evaluation criteria in the future, replacing hand-crafted criteria which cannot usually be exhaustive. In some sense, we want to extrapolate a few desirable properties into a meaningful task loss. In the future we would like to investigate how to define meaningful evaluation criteria with minimal human intervention. In this section, we describe additional advantages and properties of parametric adversarial divergences. While adversarial divergences are learned and thus potentially much more powerful than traditional divergences, the fact that they are the solution to a hard, non-convex problem can make GANs unstable. Not all adversarial divergences are equally stable: claimed that the adversarial Wasserstein gives more meaningful learning signal than the adversarial Jensen-Shannon, in the sense that it correlates well with the quality of the samples, and is less prone to mode dropping. In Section 6 we will show experimentally on a simple setting that indeed the neural adversarial Wasserstein consistently give more meaningful learning signal than the neural adversarial JensenShannon, regardless of the discriminator architecture. Similarly to the WGAN, the MMD-GAN divergence BID24 was shown to correlate well with the quality of samples and to be robust to mode collapse. Recently, it was shown that neural adversarial divergences other than the Wasserstein can also be made stable by regularizing the discriminator properly BID20 BID42. Maximum-likelihood typically requires computing the density q θ (x), which is not possible for implicit models such as GANs, from which it is only possible to sample. On the other hand, parametric adversarial divergences can be estimated with reasonable sample complexity (see Section 3.1) only by sampling from the generator, without any assumption on the form of the generator. This is also true for MMD but generally not the case for the empirical Wasserstein, which has bad sample complexity as stated previously. Another issue of f-divergences such as the Kullback-Leibler and the Jensen-Shannon is that they are either not defined (Kullback-Leibler) or uninformative (JensenShannon) when p is not absolutely continuous w.r.t. q θ BID33, which makes them unusable for learning sharp distributions such as manifolds. On the other hand, some integral probability metrics, such as the Wasserstein, MMD, or their adversarial counterparts, are well defined for any distributions p and q θ. In fact, even though the Jensen-Shannon is uninformative for manifolds, the parametric adversarial Jensen-Shannon used in the original GANs BID15 still allows learning realistic samples, even though the process is unstable BID44. Although give a lot of insights, their must be taken with a grain of salt. In this section we point out the limitations of their theory. First, their analysis ignores the dependence on x and is non-parametric, which means that they consider the whole class of possible score functions for each given x. Additionally, they only consider convex consistent surrogate losses in their analysis, and they give upper bounds but not lower bounds on the sample complexity. It is possible that optimizing approximately-consistent surrogate losses instead of consistent ones, or making additional assumptions on the distribution of the data could yield better sample complexities. C EXPERIMENTAL Here, we compare the parametric adversarial divergences induced by three different discriminators (linear, dense, and CNN) under the WGAN-GP BID18 formulation. We consider one of the simplest non-trivial generators, in order to factor out optimization issues on the generator side. The model is a mixture of 100 Gaussians with zero-covariance. The model density is q θ (x) = 1 K z δ(x − x z), parametrized by prototypes θ = (x z) 1≤z≤K. The generative process consists in sampling a discrete random variable z ∈ {1, ..., K}, and returning the prototype x z.Learned prototypes (means of each Gaussian) are shown in Figure 6 and 7. The first observation is that the linear discriminator is too weak of a divergence: all prototypes only learn the mean of the training set. Now, the dense discriminator learns prototypes which sometimes look like digits, but are blurry or unrecognizable most the time. The samples from the CNN discriminator are never blurry and recognizable in the majority of cases. Our confirms that indeed, even for simplistic models like a mixture of Gaussians, using a CNN discriminator provides a better task loss for generative modeling of images. Figure 6: Some Prototypes learned using linear (left), dense (middle), and CNN discriminator (right). We observe that with linear discriminator, only the mean of the training set is learned, while using the dense discriminator yields blurry prototypes. Only using the CNN discriminator yields clear prototypes. All 100 prototypes can be found in

Parametric adversarial divergences implicitly define more meaningful task losses for generative modeling, we make parallels with structured prediction to study the properties of these divergences and their ability to encode the task of interest.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Given Interactive technique to improve brushing in dense trajectory datasets by taking into account the shape of the brush., a positive would be Brushing techniques have a long history with the first interactive selection tools appearing in the 1990's. Since then, many additional techniques have been developed to address selection accuracy, scalability and flexibility issues. Selection is especially difficult in large datasets where many visual items tangle and create overlapping. This paper investigates a novel brushing technique which not only relies on the actual brushing location but also on the shape of the brushed area. Firstly, the user brushes the region where trajectories of interest are visible. Secondly, the shape of the brushed area is used to select similar items. Thirdly, the user can adjust the degree of similarity to filter out the requested trajectories. This technique encompasses two types of comparison metrics, the piece-wise Pearson correlation and the similarity measurement based on information geometry. We apply it to concrete scenarios with datasets from air traffic control, eye-tracking data and GPS trajectories.. This figure shows our main rationale to define our shaped based brushing technique. A) Unselected trail set where the user wishes to select the only curved one. B) Standard brushing technique, where the brushing of the curved trail also selects every other trail which touches the brushing area. C) Our method uses the brushing input to compare with the brushed trajectory and only the trajectories similar in shape are selected. Such techniques help to visually select items of interest with interactive paradigms (i.e. lasso, boxes, brush) in a view. When the user visually detects a relevant pattern (i.e. a specific curve or a trend), the brushing technique can then be applied to select it. While this selection can be seamlessly performed, the user may still face issues when the view becomes cluttered with many tangled items. In such dense visualization, existing brushing techniques also select items in the vicinity of the target and thus capture part of the clutter (see Figure 1). To address such an issue, the user can adjust the brushing parameters by changing the brush size or the selection box locations. However, this may take time and requires many iterations or trials. This paper proposes a novel brushing technique that filters trajectories taking into account the shape of the brush in addition to the brush area. This dual input is provided at the same time and opens novel opportunities for brushing techniques. The cornerstone of such a technique relies on a shape comparison algorithm. This algorithm must provide a numerical similarity measurement which is ordered (low value for unrelated shapes, and high value for correlated shapes), continuous (no steps in the computed metric) and with a semantic so that the user can partially understand the logic behind this similarity measurement. Thus, to build such a dual filtering technique, the following design requirements (DR) must be fulfilled: • DR3: The technique enables incremental selection refinement. • DR4: The technique is interactive. Taking into account the identified requirements (DR1-DR4), this paper presents a novel shape-based brushing tool. To the best of our knowledge, such a combination of brushing and shape comparison technique has not yet been explored in trajectory analysis and this paper fills this gap. In the remainder of the paper the following subjects are presented. First, previous works in the domain of brushing and shape comparison are provided. Second, the brushing pipeline is detailed with explanations of the comparison metric data processing. Next, the use of such a technique through different use-cases is demonstrated. The brushing technique is discussed in terms of usefulness, specific application and possible limitations. Finally, the paper concludes with a summary of our contribution and provides future research directions. There are various domain-specific techniques targeting trail exploration and analysis. In this section, we explore three major components of selection techniques for trail-set exploration and analysis relevant to our work: brushing, query-by-content, and similarity measurement. Trail-set exploration relies on pattern discovery where relevant trails need to be selected for further analysis. Brushing is a selection technique for information visualization, where the user interactively highlights a subset of the data by defining an area of interest. This technique has been shown to be a powerful and generic interaction technique for information retrieval. The selection can be further refined using interactive filtering techniques. The approach presented in this paper is based on dynamic queries and direct manipulation. Systems designed specifically for spatio-temporal visualization and in particular in trajectory visualizations are very complex because of their 3D and time varying nature. Due to this, several systems and frameworks have been especially designed to visualize them. Most of these systems include selection techniques based on brushing, and some of them enable further query refinement through boolean operations. These techniques do not take into account the shape of the trails, so selecting a specific one with a particular shape requires many manipulations and iterations to fine-tune the selection. While this paper attempts to suggest a shape-based brushing technique for trail sets, researchers have explored shape-based selection techniques in different contexts, both using arbitrary shapes and sketch-based queries. Sketch-based querying presents several advantages over traditional selection. It has been used for volumetric data sets, and neural pathway selection. This last work is the closest to the current study. However, the authors presented a domain-specific application and they based their algorithm on the Euclidean distance. This is not a robust metric for similarity detection since it is hard to provide a value indicating a high similarity and varies greatly according to the domain and the data considered. In addition, this metric does not support direction and orientation matching nor the combination of brushing with filtering. In addition, user-sketched pattern matching plays an important role in searching and localizing time series patterns of interest. For example, Holz and Feiner defined a relaxed selection technique in which the users draw a query to select the relevant part of a displayed time-series. Correl et al. propose a sketch-based query system to retrieve time series using dynamic time wrapping, mean square error or the Hough transform. They present all matches individually in a small multiple, arranged according to the similarity measurement. These techniques, as the one proposed here, also take advantage of sketches to manipulate data. However, they are designed for querying rather than selecting, and specifically for 2D data. Other approaches use boxes and spheres to specify the regions of interest, and the desired trails are obtained if they intersect these regions of interest. However, many parameters must be changed by the analysts in order to achieve a simple single selection. The regions of interest must be re-scaled appropriately, and then re-positioned back and forth multiple times for each operation. Additionally, many selection box modifications are required to refine the selection and thus hinder and alter the selection efficiency. Given a set of trajectories, we are interested in retrieving the most similar subset of trajectories with a user-sketched query. Common approaches include selecting the K-nearestneighbors (KNN) based on the Euclidean distance (ED) or elastic matching metrics (e.g, Dynamic Time Warping -DTWs). Affinity cues have also been used to group objects. For example, objects of identical color are given a high similarity coefficient for the color affinity. The Euclidean distance is the most simple to calculate, but, unlike mathematical similarity measurements which are usually bound between 0 and 1 or -1 and 1, ED is unbounded and task-specific. A number of works have suggested transforming the raw data into a lower-dimensional representation (e.g., SAX, PAA ). However, they require adjusting many abstract parameters which are dataset-dependant and thus reduce their flexibility. In, global and local proximity of 2D sketches are presented. The second measure is used for similarity detection where an object is contained within another one, and is not relevant to this work. While the first measure refers to the distance between two objects (mostly circles and lines), there is no guarantee that the approach could be generalized to large datasets such as eye tracking, GPS or aircraft trajectories. In contrast, Dynamic Time Warping has been considered as the best measurement in various applications to select shapes by matching their representation. It has been used in gestures recognition, eye movements and shapes. An overview of existing metrics is available. The k-Nearest Neighbor (KNN) approach has also long been studied for trail similarity detection. However, using this metric, two trails may provide a good accurate connection (i.e, a small difference measure as above) even if they have very different shapes. Other measurements to calculate trajectory segment similarity are the Minimum Bounding Rectangles (MBR) or Fréchet Distance which leverage the perpendicular distance, the parallel distance and the angular distance in order to compute the distance between two trajectories. In order to address the aforementioned issues, we propose and investigate two different approaches. The first approach is based on directly calculating the correlations on x-axis and y-axis independently between the shape of the brush and the trails (section 3.1.1). The second approach (section 3.1.2) is based on the geometrical information of the trails, i.e, the trails are transformed into a new space (using eigenvectors of the co-variance matrix) which is more suitable for similarity detection. This paper's approach leverages the potential of these two metrics to foster efficient shape-based brushing for large cluttered datasets. As such, it allows targeting detailed motif discovery performed interactively. This section presents the interactive pipeline (Figure 2) which fulfills the identified design requirements (DR1-DR4). As for any interactive system, user input plays the main role and will operate at every stage of the data processing. First, the input data (i.e. trail set) is given. Next, the user inputs a brush where the pipeline extracts the brushed items, the brush area and its shape. Then, two comparison metrics are computed between every brushed item and the shape of the brush (similarity measurement). Followed by the binning process and its filtering, the ing data is presented to the user. The user can then refine the brush area and choose another comparison metric until the desired items are selected. As previously detailed in the related word section, many comparison metrics exist. While our pipeline can use any metric that fulfill the design requirement DR2 (continuous, ordered and meaningful comparison), the presented pipeline contains only two complementary algorithms: Pearson and FPCA. The first one focuses on a shape comparison basis with correlation between their representative vertices, while the latter focuses on curvature comparison. As shown in Figure 3, each metric produces different . The user can use either of them depending on the type of filtering to be performed. During the initial development of this technique, we first considered using the Euclidean distance (ED) and DTW, but we have rapidly observed their limitations and we argue that PC and FPCA are more suitable to trajectory datasets. First, PC values are easier to threshold. A PC value > 0.8 provides a clear indication of the similarity of 2 shapes. Moreover, to accurately discriminate between complex trajectories, we need to go beyond the performance of ED. Furthermore, the direction of the trajectories, while being essential for our brushing technique, is not supported by ED and DTW similarity measures. Another disadvantage of using ED is domain-and task-specific threshold that can drastically vary depending on the context. PC, that we used in our approach, on the other hand, uses the same threshold independently of the type of datasets. The two following sections detail the two proposed algorithms. Pearson's Correlation (PC) is a statistical tool that measures the correlation between two datasets and produces a continuous measurement between ∈ [−1, 1] with 1 indicating a high degree of similarity, and −1 an anti-correlation indicating an opposite trend. This metric is well suited (DR2) for measuring dataset similarity (i.e. in trajectory points). Pearson's Correlation PC between two trails T i and T j on the x − axis can be defined as follows: Where T i x and T j x are the means, E the expectation and σ T ix, σ T jx the standard deviations. The correlation is computed on the y − axis and the x − axis for two dimensional points. This metric is invariant in point translation and trajectory scale but it does not take into account the order of points along a trajectory. Therefore, the pipeline also considers the FPCA metric that is more appropriate to trajectory shape but that does not take into account negative correlation. Functional Data Analysis is a well-known information geometry approach that captures the statistical properties of multivariate data functions, such as curves modeled as a point in an infinite-dimensional space (usually the L 2 space of square integrable functions ). The Functional Principal Component Analysis (FPCA) computes the data variability around the mean curve of a cluster while estimating the Karhunen-Loeve expansion scores. A simple analogy can be drawn with the Principal Component Analysis (PCA) algorithm where eigen vectors and their eigen values are computed, the FCPA performs the same operations with eigen functions (piece-wise splices) and their principal component scores to model the statistical properties of a considered cluster: where b j are real-valued random variables called principal component scores. φ j are the principal component functions, which obey to: φ j are the (vector-valued) eigenfunctions of the covariance operator with eigenvalues λ j. We refer the prospective reader to the work of Hurter et al. for a Discrete implementation. With this model, knowing the mean curveγ and the principal component functions φ j, a group of curves can be described and reconstructed (Inverse FPCA) with the matrix of the principal component score b j of each curve. Usually, a finite vector Figure 2. This figure shows the interaction pipeline to filtered items thanks to the user brush input. The pipeline extracts the brushed items but also the shape of the brush which will be then used as a comparison with the brushed items (metrics stage). Then, brushed items are stored in bins displayed with small multiples where the user can interactivity refine the selection (i.e. binning and filtering stage). Finally, the user can adjust the selection with additional brushing interactions. Note that both the PC and FPCA can be used for the binning process, but separately. Each small multiple comes from one metric exclusively. Figure 3. This figure shows an example of the two metrics for shape comparison usage. The user brushed around curve 3 and thus also selected curves 1 and 2. Thanks to the Pearson computation, the associated small multiples show that only curve 2 is correlated to the shape of the brush. Curve 3 is anti correlated since it in an opposite direction to the shape of the brush. The FPCA computation does not take into account the direction but rather the curvature similarity. As such, only shape 3 is considered as highly similar to the brush shape input. (with fixed dimension d) of b j scores is selected such that the explained variance is more than a defined percentile. To compute a continuous and meaningful metric (DR2), the metric computation uses the two first Principal Components (PC) to define the representative point of a considered trajectory. Then, the metric is computed by the euclidean distance between the shape of the brush and each brushed trajectory in the Cartesian scatterplot PC1/PC2 (Figure 4). Each distance is then normalized between with 1 corresponding to the largest difference in shape between the considered shape of the brush of the corresponding trajectory. Taking into account the computed comparison metrics, the pipeline stores the ing values into bins. Items can then be sorted in continuous ways from the less similar to the most similar ones. While the Pearson measurements ∈ [−1, 1] and the FPCA ∈, this binning process operates in the same way. Each bin is then used to visually show the trajectories it contains through small multiples (we use 5 small multiples which gives a good compromise between visualization compactness and trajectory visibility). The user can then interactively filter the selected items (DR4) with a range slider on top of the small multiple visualization. The user is thus able to decide whether to remove uncorrelated items or refine the correlated one with a more restricted criteria (DR3). This technique is designed to enable flexible and rapid brushing of trajectories, by both the location and the shape of the brush. The technique's interaction paradigm is now described and illustrated in a scenario where an air traffic management expert studies the flight data depicted in Figure 6. Aircraft trajectories can be visually represented as connected line segments that form a path on a map. Given the flight level (altitude) of the aircraft, the trajectories can be presented in 3D and visualized by varying their appearances or changing their representation to basic geometry types. Since the visualization considers a large number of trajectories that compete for the visual space, these visualizations often present occlusion and visual clutter issues, rendering exploration difficult. Edge bundling techniques have been used to reduce clutter and occlusion but they come at the cost of distorting the trajectory shapes which might not always be desirable. Analysts need to explore this kind of datasets in order to perform diverse tasks. Some of these tasks compare expected aircraft trajectories with the actual trajectories. Other tasks detect unexpected patterns and perform out traffic analysis in complex areas with dense traffic. To this end, various trajectory properties such as aircraft direction, flight level and shape are examined. However, most systems only support selection techniques that rely on starting and end points, or predefined regions. We argue that the interactive shape brush technique would be helpful for these kinds of tasks, as they require the visual inspection of the data, the detection of the specific patterns and then their selection for further examination. As these specific patterns might differ from the rest of the data precisely because of their shape, a technique that enables their selection through this characteristic will make their manipulation easier, as detailed in the example scenario. We consider a dataset that includes 4320 aircraft trajectories of variable lengths from one day of flight traffic over the French airspace. We define the trail T T T as a set of real-valued consecutive points T T T = [(T T T x x x 1, T T T y y y 1)), (T T T x x x 2, T T T y y y 2),..., (T T T x x x n, T T T y y y n) ] where n is the number of points and (T T T x x x i, T T T y y y i) corresponds to the i − th coordinate of the trail. The Figure 6 depicts an example of 4133 trails (aircraft in French airspace). The brush Shape consists of a set of real-valued consecutive points S S S = [(S S Sx x x 1, S S Sy y y 1)), (S S Sx x x 2, S S Sy y y 2),..., (S S Sx x x m, S S Sy y y m) ] where m is the number of points. Note that while the length n of each trail is fixed, the length m of the S S Shape depends on the length of the user brush. The S S Shape is smoothed using a 1Cfilter and then resampled to facilitate the trail comparison. The similarity metrics are then used in subsequences of the s s shape of approximately the same length as the brush S S Shape. In order to do this, each trial is first resampled so that each pair of consecutive vertices on the trail has the same distance l vertices. The user starts by exploring the data using pan and zoom operations. They are interested in the trajectories from the south-east of France to Paris. The user can choose if they are looking for a subsequence match or an exact match. A subsequence match involves the detection of trajectories having a subsequence similar to the S S Shape locally. Exact match comparison also takes into account the length of the trajectory and the S S Shape in order to select a trajectory, i.e, its length must be approximately similar to the length of the Shape (general measurements). This option is especially useful to select a trajectory by its start and end points (e.g, finding trajectories taking off from an airport A and landing at an airport B). The exact matching is supported by analyzing the length of the trail and the Shape before applying the similarity metric algorithm. The analyst in the scenario activates the subsequence match and starts brushing in the vicinity of the target trajectories following the trajectory shape with the mouse. This will define both the brush region and the brush shape. Once the brushing has been completed, the selected trajectories are highlighted in green, as depicted in Figure 5 -(b). a b Figure 5. (a) The user brushes the trajectories in order to select those from the south-east of France to Paris. (b) They select the most correlated value to take into account the direction of the trails The similarity calculation between the Shape and the brushed region will produce a similarity value for each trail contained in the region and the trails are distributed in small multiples as detailed in Section 3. Once the user has brushed, she can adjust the selection by selecting one of the bins displayed in the small multiples and using its range slider. This enables a fine control over the final selection and makes the algorithm thresholding easier to understand as the user has control over both the granularity of the exploration and the chosen similarity level. The range slider position controls the similarity level, and its size determines the number of trajectories selected at each slider position: the smallest size selects one trajectory at a time. The range slider size and position are adjusted by direct manipulation. As the bins are equally sized, the distribution of the similarity might not be linear across the small multiples. This makes navigation easier since the trajectories distribution in the small multiples is continuous. However, this also entails that not every bin will correspond to the same similarity value interval. To keep this information available to the user, a colored heatmap (from red to green) displays the actual distribution. In the current scenario, the expert, as they wish to select only the flights to Paris and not from Paris, selects the trajectories that are correlated with the original brush. These trajectories are on the right side of the small multiple, highlighted in green as depicted in Figure 5 -(b). The expert is then interested in exploring the flights that land on the north landing strip but that are not coming from the east. For this, they perform a new shape brush to identify the planes that do come from the east, and distinguishable by the "C" shape in the trajectories, as depicted in Figure 7. To be able to select the geometry precisely, the expert changes to the FPCA metric, using the keyboard shortcut. In this case the small multiple arranges the trajectories from less similar to more similar. This entails that the small multiple based on FPCA also enables the selection of all the trajectories that do not match the specified Shape but which are contained in the brushing region. As all trajectories passing through the north landing strip are contained in the brushing region, the most similar trajectories will correspond to the ones that have a "C" shape, in the same orientation as the Shape, and thus come from the east. The less similar will be the ones that interest the analyst, so they can select them by choosing the most dissimilar small multiple as depicted in Figure 7 -(b). Figure 6. One day's aircraft trajectories in French airspace, including taxiing, taking-off, cruise, final approach and landing phases. Selecting specific trajectories using the standard brushing technique will yield inaccurate due to the large number of trajectories, occlusions, and closeness in the spatial representation. We argue that there is a strong demand for targeted brushing to select motifs in datasets. In various domains, including aircraft trajectories, eye tracking, GPS trajectories or brain fiber analysis, there is a substantial need to be able to discover hidden motifs in large datasets. Undoubtedly, retrieving desired trails in such datasets would help analysts to focus on the most interesting parts of the data. The system was built using C# and OpenTK on a 64bit 1 XPS 15 Dell Laptop. Although, both PC and FPCA provide different but valuable , the running performance was 10 times faster with PC compared to FPCA. The technique was first tested informally with ML: 3? experts from aerospace domain with more than 10 years of experience in trajectories analysis. While the collected feedback was largely positive, we observed some limitations regarding the misunderstanding of our filtering parameters. Given the novelty of our interaction technique, users needed a small training period to better understand the semantic of our small multiples. Nevertheless, our technique provides initial good selection without any parameter adjustment. Because the presented technique is not designed to replace standard brushing but rather to complement it, we argue that an evaluation comparing techniques that are substantially different will not capture the actual usefulness of our technique. The goal of this technique is to facilitate trajectories selection in dense areas, where standard brushing would require multiple user actions (panning, zooming, brushing). Therefore, a use-cases approach has been chosen to showcase relevant scenarios of usage. Eye-tracking technologies are gaining popularity for analyzing human behaviour, in visualization analysis, human factors, human-computer interaction, neuroscience, psychology and training. The principle consists in finding the likely objects of interest by tracking the movements of the user's eyes. Using a camera, the pupil center position is detected and the gaze, i.e, the point in the scene the user is fixating on, is computed using a prior calibration procedure. Therefore, the gaze data consist of sampled trails representing the movements of the user's eye gaze while completing a given task. Two important types of recorded movements characterize eye behaviour: the fixations and saccades. Fixations are the eye positions the user fixates for a certain amount of time, in other words, they describe the locations that captured the attention of the user. The saccades connect the different fixations, i.e, they represent the rapid movements of the eye from one location to another. The combination of these eye movements are called the scanpath (Figure 8A). The scanpath is subject to overplotting. This challenge may be addressed through precise brushing techniques to select specific trails. Usually, fixation events are studied to create an attention map which shows the salient elements in the scene. The salient elements are located at high density fixation areas. However, the temporal connections of the different fixations provide additional information. The saccades enable the links between the fixations to be maintained and the temporal meaning of the eye movement to be held. Discovering patterns in the raw scanpath data is difficult since, in contrast to aircraft trajectories, eye movements are sparser and less regular (Figure 8). To address this, different kinds of visualizations for scanpaths have been proposed in the literature. For example, edge bundling techniques minimize visual clutter of large and occluded graphs. However, these techniques either alter trail properties such as shape and geometric information, or are otherwise computationally expensive, which makes them unsuitable for precise exploration and mining of large trail datasets. Moreover, it is possible to animate eye movements in order to have an insight of the different fixations and saccades. However, given the large datasets of eye movements retrieved from lengthy experiments containing thousands of saccades, this approach is unnecessarily time-consuming and expensive. Therefore, we next describe how this study's approach supports proper and more efficient motif discovery on such eye tracking datasets. The tested dataset is adapted from Peysakhovich et al., where a continuous recording of eye movement in a cockpit was performed. The gaze data was recorded at 50 Hz. Sequential points located in a square of 20 × 20 pixels and separated by at least 200 ms were stored as a fixation event and replaced by their average in order to reduce noise coming from the microsaccades and the tracking device. In order to illustrate some examples, we could consider a domain expert who wishes to explore the movements of the pilot's eyes in a cockpit. When performing a task, the pilot scans the different instruments in the cockpit, focuses more on certain instruments or interacts with them. Especially in this context, the order of pilot attention is important since checking a parameter in one instrument may give an indication of the information displayed in another instrument. For example, the priority of the Primary Flight Display (PFD) instrument compared to Flight Control Unit (FCU) will differ for the cruise phase as compared to the final landing approach. As an example of analysis, the user wishes to explore the movement of the eye from the Primary Flight Display (PFD) to the Navigation Display (ND). Selecting these scanpaths using traditional brushing techniques would be challenging because of the clutter, selecting those scanpaths would introduce additional accidental selections. Therefore, he brushes these scanpaths using a shape that extends from the PFD to the ND, applying the Pearson metric to consider the direction. Figure 8 (a) depicts the brushed eye movements that correspond to the most correlated trails in the small multiple. There are several saccades between those two devices, and this is inline with the fact that saccadic movements between the PFD and the ND are typically caused by parameter checking routines. However, when the user changes the selection and brushes the scanpath between the ND and the FCU, it is surprising to see that there is only one saccade between them. Brushing now with a shape that goes between the PFD and the FCU (Figure 8 -(c)) reveals only one scanpath. This is difficult to visualize in the raw data or using the standard brushing technique. A final Shape searching for an eye movement from the PFD to the LAI and passing by the FCU, in only one saccade (Figure 8-(d) ). To determine the meaning of this behavior, the tool also enables the expert to exploit a continuous transition to increase the visibility and gain insight on when these saccadic movements occurred (temporal view). The user can change the visual mapping from the (x,y) gaze location to the (time,y) temporal view. This smooth transition avoids abrupt change to the visualization (figure 9). GPS trajectories consist of sequential spatial locations recorded by a measurement instrument. Subjects such as people, wheeled vehicles, transportation modes and devices may be tracked by analyzing the spatial positions provided by these instruments. Analysts may need to explore and analyze different paths followed by the users. The advances in position-acquisition and ubiquitous devices have granted extremely large location data, which indicate the mobility of different moving targets such as autonomous vehicles, pedestrians, natural phenomena, etc. The commonness of these datasets calls for novel approaches in order to discover information and mine the data. Traditionally, researchers analyse GPS logs by defining a distance function (e.g, KNN) between two trajectories and then applying expensive processing algorithms to address the similarity detection. For example, they first convert the trajectories into a set of road segments by leveraging map-matching algorithms. Afterwards, the relationship between trajectories is managed using indexing structures. Using the data provided by Zheng et al., we seek to investigate different locations followed by the users in Beijing. The data consists of GPS trajectories collected for the Geolife project by 182 users during a period of over five years (from April 2007 to August 2012). Each trajectory is represented by a 3D latitude, longitude and altitude point. A range of users' outdoor movements were recorded, including life routines such as travelling to work, sports, shopping, etc. As the quantity of GPS data is becoming increasingly large and complex, proper brushing is challenging. Using bounding boxes somewhat alleviate this difficulty by setting the key of interest on the major corners. However, many boxes must be placed carefully for one single selection. The boxes can help the analysts to select all the trajectories that pass through a specific location, but do not simplify the analysis of overlapping and directional trajectories. This study's approach intuitively supports path differentiation for both overlapping trajectories and takes direction into account. For example, we are interested in answering questions about the activities people perform and their sequential order. For this dataset, the authors were interested in finding event sequences that could inform tourists. The shape based brushing could serve as a tool to further explore their . For example, if they find an interesting classical sequence that passes through locations A and B they can further explore if this sequence corresponds to a larger sequence and what other locations are visited before or after. A first brushing and refinement using the FPCA metric and small multiples enables them to select all the trajectories that include a precise event sequence passing through a set of locations, as depicted in Figure 10. A second brushing using the Pearson metric enables further explorations that also take into account the direction of the trajectories. Switching between the correlated trajectories and the anti-correlated ones, the user can gain insight about the visitation order of the selected locations. The proposed brushing technique leverages existing methods with the novel usage of the shape of the brush as an additional filtering parameter. The interaction pipeline shows different data processing steps where the comparison algorithm between the brushed items and the shape of the brush plays a central role. While the presented pipeline contains two specific and complementary comparison metric computations, another one can be used as long as it fulfills the continuity and metric se- Figure 10. Three different trajectories containing three different event sequences from. mantic requirements (DR2). There are indeed many standard approaches (ED, DTW, Discrete FrÃl'chet distance) that are largely used by the community and could be used to extend our technique when faced with different datasets. Furthermore, the contribution of this paper is a novel shape-based brushing technique and not simply a shape similarity measure. In our work, we found two reasonable similarity measures that fulfill our shape-based brushing method: The FPCA distance comparison provides an accurate curve similarity measurement while the Pearson metric provides a complementary criteria with the direction of the trajectory. In terms of visualization, the binning process provides a valuable overview of the order of the trajectory shapes. This important step eases the filtering and adjustment of the selected items. It is important to mention that this filtering operates in a continuous manner as such trajectories are added or removed one by one when adjusting this filtering parameter. This practice helps to fine tune the selected items with accurate filtering parameters. The presented scenario shows how small multiple interaction can provide flexibility. This is especially the case when the user brushes specific trajectories to be then removed when setting the compatibility metrics to uncorrelated. This operation performs a brush removal. The proposed filtering method can also consider other types of binning and allows different possible representations (i.e. various visual mapping solutions). This paper illustrates the shape based brushing technique with three application domains (air traffic, eye tracking, gps data), but it can be extended to any moving object dataset. However, our evaluation is limited by the number of studied application domains. Furthermore, even if various users and practitioners participated in the design of the technique, and assessed the simplicity and intuitiveness of the method, we did not conduct a more formal evaluation. The shape based brush is aimed at complementing the traditional brush, and in no way do we argue that it is more efficient or effective than the original technique for all cases. The scenarios are examples of how this technique enables the selection of trails that would be otherwise difficult to manipulate, and how the usage of the brush area and its shape to perform comparison opens novel brushing perspectives. We believe they provide strong evidence of the potential of such a technique. The technique also presents limitations in its selection flexibility, as it is not yet possible to combine selections. Many extensions can be applied to the last step of the pipeline to support this. This step mainly addresses the DR4 where the selection can be refined thanks to user inputs. As such, multiple selections can be envisaged and finally be composed. Boolean operations can be considered with the standard And, Or, Not. While this composition is easy to model, it remains difficult for an end user to master the operations when there are more than 2 subset operations. As a solution, Hurter et al. proposed an implicit item composition with a simple drag and drop technique. The pipeline can be extended with the same paradigm where a place holder can store filtered items and then be composed to produce the final . The user can then refine the selection by adding, removing or merging multiple selections. In this paper, a novel sketch-based brushing technique for trail selection was proposed and investigated. This approach facilitates user selection in occluded and cluttered data visualization where the selection is performed on a standard brush basis while taking into account the shape of the brush area as a filtering tool. This brushing tool works as follows. Firstly, the user brushes the trajectory of interest trying to follow its shape as closely as possible. Then the system pre-selects every trajectory which touches the brush area. Next, the algorithm computes a distance between every brushed shape and the shape of the brushed area. Comparison scores are then sorted and the system displays visual bins presenting trajectories from the lowest scores (unrelated -or dissimilar trajectories) to the highest values/scores (highly correlated or similar trajectories). The user can then adjust a filtering parameter to refine the actual selected trajectories that touch the brushed area and which have a suitable correlation with the shape of the brushed area. The cornerstone of this shape-based technique relies on the shape comparison method. Therefore, we choose two algorithms which provide enough flexibility to adjust the set of selected trajectories. One algorithm relies on functional decomposition analysis which insures a shape curvature comparison, while the other method insures an accurate geometric based comparison (Pearson algorithm). To validate the efficiency of this method, we show three examples of usage with various types of trail datasets. This work can be extended in many directions. We can first extend it with additional application domains and other types of dataset such as car or animal movements or any type of time varying data. We can also consider other types of input to extend the mouse pointer usage. Virtual Reality data exploration with the so called immersive analytic domain gives a relevant work extension which will be investigated in the near future. Finally, we can also consider adding machine learning to help users brush relevant trajectories. For instance, in a very dense area, where the relevant trajectories or even a part of the trajectories are not visible due to the occlusion, additional visual processing may be useful to guide the user during the brushing process.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.