Question: Represent the scientific article to find a short TLDR

Question: Deep neural networks have demonstrated unprecedented success in various knowledge management applications. However, the networks created are often very complex, with large numbers of trainable edges which require extensive computational resources. We note that many successful networks nevertheless often contain large numbers of redundant edges. Moreover, many of these edges may have negligible contributions towards the overall network performance. In this paper, we propose a novel iSparse framework and experimentally show, that we can sparsify the network, by 30-50%, without impacting the network performance. iSparse leverages a novel edge significance score, E, to determine the importance of an edge with respect to the final network output. Furthermore, iSparse can be applied both while training a model or on top of a pre-trained model, making it a retraining-free approach - leading to a minimal computational overhead. Comparisons of iSparse against PFEC, NISP, DropConnect, and Retraining-Free on benchmark datasets show that iSparse leads to effective network sparsifications. Deep neural networks (DNNs), particularly convolutional neural networks (CNN), have shown impressive success in many applications, such as facial recognition , time series analysis , speech recognition , object classification , and video surveillance (Karpathy & et. at., 2014). As the term "deep" neural networks implies, this success often relies on large networks, with large number of trainable edges (weights) (; ; ;). While a large number of trainable edges help generalize the network for complex and diverse patterns in large-scale datasets, this often comes with enormous computation cost to account for the non-linearity of the deep networks (ReLU, sigmoid, tanh). In fact, DNNs owe their recent success to hardware level innovations that render the immense computational requirements practical (Ovtcharov & et. al., 2015;). However, the benefits of hardware solutions and optimizations that can be applied to a general purpose DNN or CNN are limited and these solutions are fast reaching their limits. This has lead to significant interest in networkspecific optimization techniques, such as network compression (Choi & et. al., 2018), pruning , and regularization (Srivastava & et. al., 2014;), aim to reduce the number of edges in the network. However, many of these techniques require retraining the pruned network, leading to the significant amount of computational waste. Many successful networks nevertheless often contain large numbers of redundant edges. Consider for example, the weights of sample network shown in Figure 1a. As we see here, the weight distribution is centered around zero and has significant number of weights with insignificant contribution to the network output. Such edges may add noise or non-informative information leading to reduction in the network performance. (; ;) has shown that it is possible to predict 95% network parameters while only learning 5% parameters. Sparsification techniques can generally be classified into neuron/kernel sparcification and edge/weight sparcification techniques (; Ashouri et al., Figure 1 : Overview of weight distribution and model accuracies for MNIST dataset 2018): proposed to eliminate neurons that have low l2-norms of their weights, whereas proposed a neuron importance score propagation (NISP) technique where neuron importance scores (using Roffo & et. al. -See Equation 5 ) are propagated from the output layer to the input layer in a back-propagation fashion. Drop-out (Srivastava & et. al., 2014) technique instead deactivates neuron activations at random. As an edge sparsification technique, DropConnect selects edges to be sparsified randomly. showed that the network performance can be maintained by eliminating insignificant weights without modifying the network architecture. Following these works, we argue network sparsification can be a very effective tool for reducing sizes and complexities of DNNs and CNNs, without any significant loss in accuracy. However, we also argue that edge weights cannot be used "as is" for pruning the network. Instead, one needs to consider the significance of each edge within the context of their place in the network (Figure 2): "Two edges in a network with the same edge weight may have different degrees of contributions to the final network output" and in this paper, we show that it is possible to quantify significance of each edge in the network, relative to their contributions to the final network output and use these measures significance to minimize the redundancy in the network by sparsifying the weights that contributes insignificantly to network. We, therefore, propose a novel iSparse framework, and experimentally show, that we can sparsify the network, by almost 50%, without impacting the network performance. The key contributions of our proposed work are as follows: • Output-informed quantification of the significance of network parameters: Informed by the final layer network output, iSparse computes and propagates edge significant scores that measure the importance of each edge with respect to the model output (Section 3). • Retraining-free network sparsification (Sparsify-with): The proposed iSparse framework is robust to edge sparsification and can maintain network performance without having to retraining the network. This implies that one can apply iSparse on pre-trained networks, on-the-fly, to achieve the desired level of sparsification (Section 3.3) • Sparsification during training (Train-with): iSparse can also be used as a regularizer during the model training allowing for learning of sparse networks from scratch (Section 4). As the sample in Figure 1b shows, iSparse is able to achieve 30-50% sparsification with minimal impact on model accuracy. More detailed experimental comparisons (See Section 5) of iSparse against PFEC, NISP, Retraining-Free and DropConnect on benchmark datasets illustrated that iSparse leads to more effective network sparsifications. A neural network is a sequence of layers of neurons to help learn (and remember) complex nonlinear patterns in a given dataset . Recently, deep neural networks (DNNs), and particularly CNNs, which leverage recent hardware advances to increase the number of layers in the -.*,)$#*)#*/0#)10+%22#).*3. * Figure 2: Overview of iSparse sparsification, considering the n i's contribution to overall output rather than only between n i and n j neurons network to scales that were not practical until recently, (; ; ; ; Karpathy & et. at., 2014) have shown impressive success in several data analysis and machine learning applications. A typical CNN consists of feature extraction layers are responsible for learning complex patterns in the data and remember them through layer weights., by training for a weight matrix W l R m l ×n l (see Section 3.1 for further details); activation layers, which help capture non-linear patterns in the data through activation functions (σ) which maps the output from a feature extraction layer to a nonlinear space (ReLU and softmax are commonly used activation functions); and pooling layers, (sampling) which up-or down-sample the intermediate data in the network. The training process of a neural network often comprises of two key stages: forward-propagation (upstream) maps the input data, X, to an output variable,Ŷ. At each layer, we haveŶ The number of trainable parameters in a deep network can range from as low as tens of thousands to hundreds of millions (Table 1 in Section 5). The three order increase in the number trainable parameter may lead to parameters being redundant or may have negligible contribution to the overall network output. This redundancy and insignificance of the network parameters has led to advancements in network regularization, by introducing dynamic or informed sparsification in the network. These advancements can be broadly classified into two main categories: parameter pruning and parameter regularization. In particular, pruning focuses on compressing the network by eliminating the redundant or insignificant parameters. (; Han & et. al., 2016) aims to prune the parameters with near-zero weights inspired from l 1 and l 2 regularization . choose to filter out convolutional kernel with minimum weight values in given layer. Recently, minimizes the change in final network performance by eliminating the neuron that have minimal impact on the network output by leveraging neuron importance score (N L) (See Section 5.3) -computed using Inf-FS (Roffo & et. al., 2015). More complex approaches have been proposed to tackle the problem of redundancy in the network through weight quantization. (Rastegari & et. al., 2016) propose to the quantize the inputs and output activations of the layers in a CNN by using step function and also leveraging the binary operation by using the binary weights opposed to the real-values weights. (Chen & et. al., 2015) focuses on low-level mobile hardware with limited computational power, and proposed to leverage the inherent redundancy in the network for using hashing functions to compress the weights in the network. showed that the each input feature to a given layer in the network rarely have the same importance, therefore, learning there individual importance (attention) helps improve the performance of the network. More recently, has shown that input data informed deep networks can provide high-performance network configurations. In this paper, we rely on output information for identifying and eliminating insignificant parameters from the network, without having to update the edge weights or retraining the network. As discussed in Section 1, in order to tackle complex inputs, deep neural networks have gone increasingly deeper and wider. This design strategy, however, often in large numbers of insignificant edges 1 (weights), if not redundant. In this section, we describe the proposed iSparse, framework which quantifies the significance of each individual connection in the network with respect to the overall network output to determine the set of edges that can be sparsified to alleviate network redundancy and eliminate insignificant edges. iSparse aims to determine the significance of the edges in the network to make informed sparsification of the network. A typical neural network, N, can be viewed as a sequential arrangement of convolutional (C) and fully connected (F) layers:, here, X is the input, L is the total number of layers in the network and L {C, F}, s.t., any given layer, where, Y l is the input to the layer (s.t. Y l =Ŷ l−1 and for l = 1, Y 1 = X) and σ l, W l, and B l are the activation function, weight, and bias respectively. Note that, if the l th layer has m l neurons and the (l − 1) th layer has n l neurons, Given this formulation, the problem of identifying insignificant edges can be formulated as the problem of generating a sequence of binary mask matrices, M 1,..., M L, that collectively represents whether any given edge in the network is sparsified or not: Let M l be a mask matrix as defined in Equation 2, and M l can be expanded as where each M l,i,j ∈ {0, 1} corresponds to an edge e l,i,j in the network. Our goal in the paper is to develop an edge significant score measure to help set the binary value of M l,i,j for each edge in the network. More specifically, we aim to associate a non-negative real valued number, E l,i,j ≥ 0, to each edge in the network, s.t. Here, τ l (θ l) represents the lowest significance of the θ l % of the most significant edges in the layer l. Intuitively, given a target sparsification rate, θ l, we rank all the edges based on their edge significance scores and keep only the highest scoring θ l % of the edges by setting their mask values to 1. As we have seen in Figure 1a, the (signed) weight distribution of the edges in a layer is often centered around zero, with large numbers of edges having weights very close to 0. As we also argued in the Introduction, such edges can work counter-intuitively and add noise or non-informative information leading to reduction in the network performance. In fact, several existing works, such as , relies on these weights for eliminating insignificant edges without having to retrain the network architecture. However, as we also commented in the Introduction, we argue that edge weights should not be used alone for sparsifying the network. Instead, one needs to consider each edge within the context of their place in the network: Two edges in a network with the same edge weight may have different degrees of contributions to the final network output. Unlike existing works, iSparse takes this into account when selecting the edges to be sparsified (Figure 3). Figure 3: A sample network architecture and its sparsification using Retraining-free and iSparse; here node labels indicate input to the node; edge labels indicate the edge weights; and edge labels between parentheses indicate edge contribution More specifically, let W + l be the absolute positive of the weight matrix,W l, for edges in l th layer. We compute the corresponding edge significance score matrix, E l, as where, N l represents the neuron significance scores 2, N l,1 through N l,n l, and " " represents the scalar multiplication between edge weights and neuron scores. N l,i, denotes the significance of the i th input neuron to the l th connection layer of the network, which itself is defined recursively, based on the following layer in the network, using the conventional dot product: Note that N l can be expanded as Above, N L denotes the neuron scores of the final output layer, and N L is defined using infinite feature selection (Roffo & et. al., 2015;) as x is the number of input samples and n is the number of output neurons) to determine neuron importance score with respect to the the final network output. Given the above, the edge score (Equation 5) can be rewritten as Note that the significance scores of edges in layer l considers not only the weights of the edges, but also the weights of all downstream edges between these edges and the final output layer. As noted in Section 3.1, the binary values in the masking matrix M l depends on τ l (θ l), which represents the lowest significance of the θ l % of the most significant edges in the layer 3: therefore, given a target sparsification rate, θ l, for layer l, we rank all the edges based on their edge significance scores and keep only the highest scoring θ l % of the edges by setting their mask values to 1. Note that, once an edge is sparsified, change in its contribution is not propagated back to the layers earlier in the network relative to the sparsified edge. Having determined the insignificant edges with respect to the final layer output, represented in form of the mask matrix, M l (described in Section 3.1), the next step is to integrate this mask matrix in the layer itself. To achieve this, iSparse extends the layer l (Equation 1) to account for the corresponding mask matrix (M l): where, * represents the element-wise multiplication between the matrices W l and M l. Intuitively, M l facilitates introduction of informed sparsity in the layer by eliminating edges that do not contribute significantly to the final output layer. In the previous section, we discussed the computation of edge significance scores on a pre-trained network, such as of pre-trained ImageNet models, and the use of these scores for network sparsification. In this section, we highlight that iSparse can also be integrated directly within the the training process. To achieve this, the edge significance score is computed for every trainable layer in the network using the strategy described in Section 3.2 and the mask matrix is updated using Equation 4. Furthermore, the back-propagation rule, described in Section 2, is updated to account for the mask matrices: where, W l are the updated weights, W l original weights, η is the learning rate, and Err l is the error recorded by as the divergence in between ground truth (Y l) and model predictions (Ŷ l) as Note that, we argue that any edge that does not contribute towards the final model output, must not be included in the back-propagation. Therefore, we mask the error as Err l * M l. In this section, we experimentally evaluate of the proposed iSparse framework using LeNet and VGG architectures (See Section 5.2) and compare it against the approaches, such as PFEC, NISP, and DropConnect (see Section 5.3). We implemented iSparse in Python environment (3.5.2) using Keras Deep Learning Library (2.2.4-tf) with TensorFlow Backend (1.14.0). All experiments were performed on an Intel Xeon E5-2670 2.3 GHz Quad-Core Processor with 32GB RAM equipped with Nvidia Tesla P100 GPU with 16 GiB GDDR5 RAM with CUDA-10.0 and cuDNN v7.6.4. 4. In this paper, without loss of generality, we leverage LeNet-5 and VGG-16 as the baseline architectures to evaluate sparsification performance on different benchmark image classification datasets and for varying degrees of edge sparsification. In this section, we present an overview of these architectures (See Table 1). LeNet-5: Designed for recognizing handwritten digits, LeNet-5 is simple network with 5 trainable (2 convolution and 3 dense) and 2 non-trainable layers using average pooling with tanh and softmax as the hidden and output activation. LeNet's simplicity has made it a common benchmark for datasets recorded in constrained environments, such as MNIST , FMNIST , COIL (a; b), and NORB . VGG-16: VGG 's, a 16 layer network with 13 convolution and 3 dense layers, with interleaved 5 max-pooling layers. VGG leverages ReLU as the hidden activation to overcome the problem of vanishing gradient, as opposed to tanh. Given the ability of VGG network to learn the complex pattern in the real-world dataset, we use the network on benchmark datasets, such as CIFAR10/20/100 , SVHN (Netzer & et. al., 2011), GTSRB (Stallkamp & et. al., 2012), and ImageNet (Deng et. al., 2009). Table 1 reports the number of trainable parameters (or weights) for each model/data set pair considered in the experiments. We compared iSparse against several state-of-the-art network sparsification techniques: DropConnect is a purely random approach, where edges are randomly selected for sparsification. Retraining-free considers each layer independently and sparsifies insignificant weights in the layer, without accounting for the final network output contribution. PFEC is a kernel pruning strategy that aims to eliminate neurons that have low impact on the overall model accuracy. In order to determine the impact, PFEC computes the l2-norms of the weights of the neurons and ranks them, separately, for each layer. NISP proposes a neuron importance score propagation (NISP) technique where neuron importance scores are propagated from the output layer to the input layer in a back-propagation fashion. Figure 7: Mask matrices for the LeNet network conv 2 layer for MNIST data (sparsification factor = 50%): dark regions indicate the edges that have been marked for sparsification; in (e) iSparse, the arrows point to those edges that are subject to different pruning decision from retrain-free in(d) (green arrows point to edges that are kept in iSparse instead of being pruned and red arrows point to edges that are sparsified in iSparse instead of being kept) In Figure 4, we first present top-1 and top-5 classification for ImageNet dataset for VGG-16 network. As we see in the Figure 4, iSparse provides the highest robustness to the degree of sparsification in the network. In particular, with iSparse, the network can be sparsified by 50% with ≤ 6% drop in accuracy for top-1 and ≤ 2% drop in accuracy for top-5 classification, respectively. In contrast, the competitors, see larger drops in accuracy. The closest competitor, Retrain-free, suffers a loss in accuracy of ∼ 16% and ∼ 6% for top-1 and top-5 classification, respectively. The other competitors suffer significant accuracy drops after a mere 10-20% sparsification. Figures 6a and 6b show the top-1 classification accuracy for other models and data sets. As we see here, the above pattern holds for all configurations considered: iSparse provides the best robustness. It is interesting to note that DropConnect, NISP, and PFEC see especially drastic drops in accuracy for the VGG-16 network and especially on the CIFAR data. This is likely because, VGG-CIFAR is already relatively sparse (20% > sparsity as opposed to ∼ 7% for VGG-ImageNet and < 1% for LeNet) and these three techniques are not able to introduce additional sparseness in a robust manner. In contrast, iSparse is able to introduce significant additional sparsification with minimal impact on accuracy. Figure 7 provides the mask matrices created by the different algorithms to visual illustrate the key differences between the competitors. As we see in this figure, PFEC and NISP, both sparsify input neurons. Consequently, their effect is to mask out entire columns from the weight matrix and this prevents these algorithms to provide fine grain adaption during sparsification. DropConnect selects individual edges for sparsification, but only randomly and this prevents the algorithm to provide sufficiently high robustness. Retrain-free and iSparse both select edges in an fine-grained manner: retrain-free uses relies on edge-weights, whereas iSparse complements edge-weight with an edge significance measure that accounts for each edges contribution to the final output within the overall network. As we see in Figure 7 (d) and (e), this in some differences in the corresponding mask matrices, and these differences are sufficient to provide significant boost in accuracy. Tables 2 present accuracy for the scenarios where iSparse (iS) is used to sparsify the model during the training process. The table also considers DropConnect (DC) and Retrain-Free (RF), as alternatives. As we see in the table, for both network architectures, under most sparsification rates, the output informed sparsification approach underlying iSparse leads to networks with the highest classification accuracies. In this section, we study the effect of the variations in network elements. In particular, we compare the performance of iSparse (iS) against DropConnect (DC) and Retraining-Free (RF) for different hidden activation functions and network optimizers. Table 3 presents classification performances for networks that rely on different activation functions (tanh and ReLU) and for optimizers (Adam and RMSProp). As we see in these two tables, iSparse remains the alternative which provides the best classification accuracy under different activation/optimization configurations. We next investigate the performance of iSparseunder different orders in which the network layers are sparsified. In particular, we considerthree sparsification orders: (a) input-to-output layer order: this is the most intuitive approach as it does not require edge significance scores to be revised based on sparsified edges in layers closer to the input; (b) output-to-input layer-order: in this case, edges in layers closer to the network output are sparsified first -but, this implies that edge significance scores are updated in the earlier layers in the network to account for changes in the overall edge contributions to the network; (c) random layer order: in this case, to order of the layers to be sparsified is selected randomly. Figure 8 presents the sparsification for different orders, data sets, and sparsification rates. As we see in the figure, the performance of iSparse is not sensitive to the sparsification order of the network layers. In Figure 5, we investigate the impact of edge sparsification on the classification time. As we see in this Figure, edge sparsification rate has a direct impact on the classification time of the ing model. When we consider that iSparse allows for ∼ 30 − 50% edge sparsification without any major impact on classification accuracies, this indicates that iSparse has the potential to provide significant performance gains. What is especially interesting to note in Figure 5 is that, while all three sparsification methods, iSparse, DropConnect, and Retraining-Free, all have the same number of sparsified edges for a given sparsification factor, the proposed iSparse approach leads to the least execution times among the three alternatives. We argue that this is because the output informed sparsification provided by iSparse allows for more efficient computations in the sparsified space 5. In this paper, we proposed iSparse, a novel output-informed, framework for edge sparsification in deep neural networks (DNNs). In particular, we propose a novel edge significance score that quantifies the significance of each edge in the network relative to its contribution to the final network output. iSparse leverages this edge significance score to minimize the redundancy in the network by sparsifying those edges that contribute least to the final network output. Experiments, with 11 benchmark datasets and using two well-know network architectures have shown that the proposed iSparse framework enables 30 − 50% network sparsification with minimal impact on the model classification accuracy. Experiments have also shown that the iSparse is highly robust to variations in network elements (activation and model optimization functions) and that iSparse provides a much better accuracy/classification-time trade-off against competitors.

represent this text to find a much much longer article about it with say hundreds-thousands of words

To give you a sense - "We propose a generative neural network approach for temporally coherent point clouds." should be close to "Point clouds, as a form of Lagrangian representation, allow for powerful and flexible applications in a large number of computational disciplines. We propose a novel deep-learning method to learn stable and temporally coherent feature spaces for points clouds that change over time. We identify a set of inherent problems with these approaches: without knowledge of the time dimension, the inferred solutions can exhibit strong flickering, and easy solutions to suppress this flickering can in undesirable local minima that manifest themselves as halo structures. We propose a novel temporal loss function that takes into account higher time derivatives of the point positions, and encourages mingling, i.e., to prevent the aforementioned halos. We combine these techniques in a super-resolution method with a truncation approach to flexibly adapt the size of the generated positions. We show that our method works for large, deforming point sets from different sources to demonstrate the flexibility of our approach. Deep learning methods have proven themselves as powerful computational tools in many disciplines, and within it a topic of strongly growing interest is deep learning for point-based data sets. These Lagrangian representations are challenging for learning methods due to their unordered nature, but are highly useful in a variety of settings from geometry processing and 3D scanning to physical simulations, and since the seminal work of , a range of powerful inference tasks can be achieved based on point sets. Despite their success, interestingly, no works so far have taken into account time. Our world, and the objects within it, naturally move and change over time, and as such it is crucial for flexible point-based inference to take the time dimension into account. In this context, we propose a method to learn temporally stable representations for point-based data sets, and demonstrate its usefulness in the context of super-resolution. An inherent difficulty of point-based data is their lack of ordering, which makes operations such as convolutions, which are easy to perform for Eulerian data, unexpectedly difficult. Several powerful approaches for point-based convolutions have been proposed (; ;), and we leverage similar neural network architectures in conjunction with the permutation-invariant Earth Mover's Distance (EMD) to propose a first formulation of a loss for temporal coherence. In addition, several works have recognized the importance of training point networks for localized patches, in order to avoid having the network to rely on a full view of the whole data-set for tasks that are inherently local, such as normal estimation , and super-resolution ). This also makes it possible to flexibly process inputs of any size. Later on we will demonstrate the importance of such a patch-based approach with sets of changing cardinality in our setting. A general challenge here is to deal with varying input sizes, and for super-resolution tasks, also varying output sizes. Thus, in summary we target an extremely challenging learning problem: we are facing permutation-invariant inputs and targets of varying size, that dynamically move and deform over time. In order to enable deep learning approaches in this context, we make the following key contributions: Permutation invariant loss terms for temporally coherent point set generation; A Siamese training setup and generator architecture for point-based super-resolution with neural networks; Enabling improved output variance by allowing for dynamic adjustments of the output size; The identification of a specialized form of mode collapse for temporal point networks, together with a loss term to remove them. We demonstrate that these contributions together make it possible to infer stable solutions for dynamically moving point clouds with millions of points. More formally, we show that our learning approach can be used for generating a point set with an increased resolution from a given set of input points. The generated points should provide an improved discretization of the underlying ground truth shape represented by the initial set of points. For the increase, we will target a factor of two to three per spatial dimension. Thus, the network has the task to estimate the underlying shape, and to generate suitable sampling positions as output. This is generally difficult due to the lack of connectivity and ordering, and in our case, positions that move over time in combination with a changing number of input points. Hence it is crucial that the network is able to establish a temporally stable latent space representation. Although we assume that we know correspondences over time, i.e., we know which point at time t moved to a new location at time t + ∆t, the points can arbitrarily change their relative position and density over the course of their movement, leading to a substantially more difficult inference problem than for the static case. Deep learning with static point sets was first targeted in PointNet via order-invariant networks, while PointNet++ (extended this concept to generate features for localized groups similar to a convolution operation for grid-based data. This concept can be hierarchically applied to the generated groups, in order to extract increasingly abstract and global features. Afterwards, the extracted features can be interpolated back to the original point cloud. The goal to define point convolutions has been explored and extended in several works. The MCNN approach phrased convolution in terms of a Monte Carlo integration. PointCNN defined a pointwise convolution operator using nearest neighbors, while extension-restriction operators for mapping between a point cloud function and a volumetric function were used in. The PU-Net proposed a network for upsampling point clouds, and proposed a similar hierarchical network structure of PointNets along the lines of PointNet++ to define convolutions. Being closer to our goals, we employ this approach for convolutional operations in our networks below. We do not employ the edge-aware variant of the PU-Net (b) here, as we focus on temporal changes in our work. Permutation invariance is a central topic for point data, and was likewise targeted in other works . The Deep Kd-network defined a hierarchical convolution on point clouds via kd-trees. PointProNets employed deep learning to generate dense sets of points from sparse and noisy input points for 3D reconstruction applications. PCPNet , as another multi-scale variant of PointNet, has demonstrated high accuracy for estimating local shape properties such as normal or curvature. P2PNet ) used a bidirectional network and extends PointNet++ to learn a transformation between two point clouds with the same cardinality. Recently, the area of point-based learning has seen a huge rise in interest. One focus here are 3D segmentation problems, where numerous improvements were proposed, e.g., by SPLATNet , SGPN (a), SpiderCNN , PointConv , SONEt and 3DRNN . Other networks such as Flex Convolution , the SuperPoint Graph , and the fully convolutional network focused on large scale segmentation. Additional areas of interest are shape classification (b; ; ;) and object detection , and hand pose tracking . Additional works have targeted rotation and translation invariant inference , and point cloud autoencoders ). A few works have also targeted generative models based on points, e.g., for point cloud generation, and with adversarial approaches. It is worth noting here that despite the huge interest, the works above do not take into account temporally changing data, which is the focus of our work. A notable exception is an approach for scene flow, in order to estimate 3D motion directly on the basis of point clouds. This work is largely orthogonal to ours, as it does not target generative point-based models. We assume an input point cloud, where d includes 3 spatial coordinates and optionally additional features. Our goal is to let the network f s (X) infer a functionỸ which approximates a desired super-resolution output point cloud Y = {y 1, y 2, ..., y n} of size n ∈ [1, n max] with y i ∈ R 3, i.e. f s (X) =Ỹ ≈ Y. For now we assume that the number of output points n is defined by multiplying k with a user-defined upsampling factor r, i.e. n = rk. Figure 2a) illustrates the data flow in our super-resolution network schematically. We treat the upsampling problem as local one, i.e., we assume that the inference problem can be solved based on a spatially constrained neighborhood. This allows us to work with individual patches extracted from input point clouds. At the same time, it makes it possible to upsample adaptively, for example, by limiting the inference to relevant areas, such as complex surface structures. For the patch extraction we use a fixed spatial radius and normalize point coordinates within each patch to lie in the range of [−1, 1]. Our first building block is a measure for how well two point clouds represent the same object or scene by formulating an adequate spatial loss function. , we base our spatial loss L S on the Earth Mover's Distance (EMD), which solves an assignment problem to obtain a differentiable bijective mapping φ:ỹ → y. With φ we can minimize differences in position for arbitrary orderings of the points clouds via: Figure 2: a) Schematic overview of fs(X). Black arrows represent scalar data. Point data is depicted as colored arrows with the color indicating data cardinality (brown=k, red = kmax, green = nmax, blue = n, and purple =ñ). b) Siamese network setup for temporal loss calculation. When not taking temporal coherence explicitly into account, the highly nonlinear and ill-posed nature of the super-resolution problem can cause strong variations in the output even for very subtle changes in the input. This in significant temporal artifacts that manifest themselves as flickering. In order to stabilize the output while at the same time keeping the network structure as small and simple as possible, we propose the following training setup. Given a sequence of high resolution point clouds Y t, with t indicating time, we can compute a velocity For this we use a finite difference (y t+1 i − y t i), where we assume, without loss of generality, ∆t = 1, i.e. the time step is normalized to one. For training, the low resolution inputs X can now be generated from Y via down-sampling by a factor of r, which yields a subset of points with velocities. Details of our data generation process will be given below. To train a temporally coherent network with the Y t sequences, we employ a Siamese setup shown in Figure 2b. We evaluate the network several times (3 times in practice) with the same set of weights, and moving inputs, in order to enforce the output to behave consistently. In this way we avoid recurrent architectures that would have to process the high resolution outputs multiple times. In addition, we can compute temporal derivatives from the input points, and use them to control the behavior of the generated output. Under the assumption of slowly moving inputs, which theoretically could be ensured for training, a straightforward way to enforce temporal coherence would be to minimize the generated positions over consecutive time steps in terms of an L 2 norm: While this reduces flickering, it does not constrain the change of velocities, i.e., the acceleration. This in a high frequency jittering of the generated point positions. The jitter can be reduced by also including the previous state at time step t − 1 to constrain the acceleration in terms of its L 2 norm: However, a central problem of a direct temporal constraint via Equations and is that it consistently leads to a highly undesirable clustering of generated points around the center point. This is caused by the fact that the training procedure as described so far is unbalanced, as it only encourages minimizing changes. The network cannot learn to reconstruct realistic, larger motions in this way, but rather. can trivially minimize the loss by contracting all outputs to a single point. For this reason, we instead use the estimated velocity of the ground truth point cloud sequence with a forward difference in time, to provide the network with a reference. By using the EMD-based mapping φ established for the spatial loss in Equation, we can formulate the temporal constraint in a permutation invariant manner as Intuitively, this means the generated outputs should mimic the motion of the closest ground truth points. As detailed for the L 2 -based approaches above, it makes sense to also take the ground truth acceleration into account to minimize rapid changes of velocity over time. We can likewise formulate this in a permutation invariant way w.r.t. ground truth points via: We found that a combination of L EV and L EA together with the spatial loss L S from Eq. 1 provides the best , as we will demonstrate below. First, we will introduce the additional loss terms of our algorithm. Existing network architectures are typically designed for processing a fixed amount of input and output points. However, in many cases, and especially for a localized inference of super-resolution, the number of input and output points varies significantly. While we can safely assume that no patch exceeds the maximal number of inputs k max (this can be ensured by working on a subset), it can easily happen that a certain spatial region has fewer points. Simply including more distant points could guarantee that we have a full set of samples, but this would mean the network has to be invariant to scaling, and to produce features at different spatial scales. Instead, we train our network for a fixed spatial size, and ensure that it can process varying numbers of inputs. For inputs with fewer than k max points, we pad the input vector to have a fixed size. Here, we ensure that the padding values are not misinterpreted by the network as being point data. Therefore, we pad X with p ∈ {−2} d, which represents a value outside the regular patch coordinate range [−1, 1]: The first convolutional layer in our network now filters out the padded entries using the following mask: The entries of p allow us to compute the mask on the fly throughout the whole network, without having to pass through k. For an input of size k, our network has the task to generateñ = rk points. As the size of the network output is constant with rk max, the outputs are likewise masked with M out to truncate it to lengthñ for all loss calculations, e.g., the EMD mappings. Thus, as shown in Figure 2a,ñ is used to truncate the point cloudȲ = {ȳ 1,ȳ 2, ...,ȳ nmax} via a mask M out to form the final output Note that in Sec. 3.1, we have for simplicity assumed that n = rk, however, in practice the number of ground truth points n varies. As such,ñ only provides an approximation of the true number of target points in the ground truth data. While the approximation is accurate for planar surfaces and volumes, it is less accurate in the presence of detailed surface structures that are smaller than the spatial frequency of the low-resolution data. We have analyzed the effect of this approximation in Fig. 3. The histograms show that the strongly varying output counts are an important factor in practice, and Fig. 4 additionally shows the improvement in terms of target shape that from incorporating variable output sizes. In general,ñ provides a good approximation for our data sets. However, as there is a chance to infer an improved estimate of the correct output size based on the input points, we have experimented with training a second network to predictñ in conjunction with a differentiable M out. While this could be an interesting feature for future applications, we have not found it to significantly improve . As such, the evaluations and below will use the analytic calculation, i.e.,ñ = rk. For each input point the network generates r output points, which can be seen as individual groups g: ψ(g) = {Ỹ i |i ∈ [rg + 1, (r + 1)g]}. These groups of size r in the output are strongly related to the input points they originate from. Networks that focus on maintaining temporal coherence for the dynamically changing output tends to slide into local minima where r output points are attached as a L2 Loss Velocity Only Velocity + Acceleration Figure 6: Ablation study for our temporal loss formulation. Black points indicate targets, while green points are generated (both shown as time average). a) Result from previous work; b) With L2V loss; c) the proposed velocity loss LEV; d) our full loss formulation with LEV + LEA. While (a) has difficulties to approximate the target shape and the flickering output is visible as blurred positions, the additional loss terms (esp. in (c) and (d) Table 1: Quantitative for the different terms of our loss functions, first for our 2D ablation study and then for our 3D versions. The first three columns contain spatial, the next four temporal metrics. LN = ñ − n 2 2 is given as a measure of accuracy in terms of the size of the generated outputs (it is not part of the training). fixed structure to the input point location. This manifests itself as visible halo-like structures that move statically with the input. Although temporal coherence is good in this case, these cluster-like structures lead to gaps and suboptimal point distributions in the output, particularly over time. These structures can be seen as a form of temporal mode collapse that can be observed in other areas deep learning, such as GANs. To counteract this effect, we introduce an additional mingling loss term to prevent the formation of clusters by pushing the individual points of a group apart: Figure 5: Left, a without the mingling loss from Eq. 6, right with (a single point group highlighted in orange). The former has many repeated copies of a single pattern, which the mingling manages to distribute. Note that in contrast to previously used repulsion losses, L M encourages points to globally mix rather than just locally repelling each other. While a repulsion term can lead to a deterioration of the generated outputs, our formulation preserves spatial structure and temporal coherence while leading to well distributed points, as is illustrated in Fig. 5. In combination with the spatial and temporal terms from above, this leads to our final loss function L f inal = L S + γL EV + µL EA + νL M, with weighting terms γ, µ, ν. We train our network in a fully supervised manner with simulated data. To illustrate the effect of our temporal loss functions, we employ it in conjunction with established network architectures from previous work (; . Details of the data generation and network architectures are given in the appendix. We first discuss our data generation and training setup, then illustrate the effects of the different terms of our loss function, before showing for more complex 3D data sets. As our focus on temporal coherence, which is best seen a) b) c) Figure 7: Our method applied to an animation of a moving spider. (a) Input point cloud, (b) three frames of our method, (c) a detail from previous work (top) and our method (bottom). Note that our method at the bottom preserves the shape with fewer outliers, and leads to a more even distribution of points, despite generating fewer points in total (see Table 2). a) shows averaged latent space values for 100 random patch sequences of our 2D data set. The green curve shows our method with temporal coherence loss, while the pink curve was generated without it. The same data is shown in frequency space in (b), where the red curve represents the frequency of the data with temporal loss, and the blue curve the frequency of the data without. These graph highlights the reduced amount of high frequency changes in the latent space with temporal loss, esp. in frequency space, where the red curve almost entirely lies below the blue one. (c) contains frequency information for the latent space content of the same 100 patch sequences, but with a random order. In this case, the blue and red curve both contain significant amounts of high-frequencies. I.e., our method reliably identifies strongly changing inputs. Table 2: Point counts for the 3D examples of our video. Input counts together with output counts for previous work (P.W.) and our proposed network are shown. Factor columns contain increase in point set size from in-to output. As previous work cannot handle flexible output counts, a fixed number of points is generated per patch, leading to a huge number of redundant points. However, our network flexibly adapts the output size and leads to a significantly smaller number of generated points that cover the object or volume more evenly. in motion, we refer readers to the supplemental video at https://www.dropbox.com/sh/ btrzxavn34qftfe/AAADpIBME0eguA4ew4ylvCX_a?dl=0 in order to fully evaluate the ing quality. Ablation Study We evaluate the effectiveness of our loss formulation with a two dimensional ablation study. An exemplary patch of this study is shown in Fig. 6. In order to compare our method to previous work, we have trained a previously proposed method for point-cloud super-resolution, the PU-Net ) which internally uses a PointNet++ , with our data set, the only difference being that we use zero-padding here. This architecture will be used in the following comparisons to previous work. Fig. 6a ) shows a generated with this network. As this figure contains an average of multiple frames to indicate temporal stability, the blurred regions, esp. visible on the right side of Fig. 6a ), indicate erroneous motions in the output. For this network the difficulties of temporally changing data and varying output sizes additionally lead to a suboptimal approximation of the target points, that is also visible in terms of an increased L S loss in Table 1. While Fig. 6b ) significantly reduces motions, and leads to an improved shape as well as L S loss, its motions are overly constrained. E.g., at the bottom of the shown patch, the generated points should follow the black input points, but in (b) the generated points stay in place. In addition, the lack of permutation invariance leads to an undesirable clustering of generated points in the patch center. Both problems are removed with L EV in Fig. 6c ), which still contains small scale jittering motions, unfortunately. These are removed by L EA in Fig. 6d), which shows the of our full algorithm. The success of our approach for dynamic output sizes is also shown in in the L N column of Table 1, which contains an L 2 error w.r.t. ground truth size of the outputs. Temporally Coherent Features A central goal of our work is to enable the learning of features that remain stable over time. To shed light on how our approach influences the established latent space, we analyze its content for different inputs. The latent space in our case consists of a 256-dimensional vector that contains the features extracted by the first set of layers of our network. Fig. 8 contains a qualitative example for 100 randomly selected patch sequences from our test data set, where we collect input data by following the trajectory of each patch center for 50 time steps to extract coherent data sets. Fig. 8a ) shows the averaged latent space content over time for these sequences. While the model trained with temporal coherence (green curve) is also visually smoother, the difference becomes clearer when considering temporal frequencies. We measure averaged frequencies of the latent space dimensions over time, as shown in Fig. 8b,c). We quantify the differences by calculating the integral of the frequency spectrumf, weighted by the frequency x to emphasize high frequencies, i.e, x x ·f (x)dx. Hence, small values are better. As shown in Fig. 8b ), the version trained without our loss formulations contains significantly more high frequency content. This is also reflected in the weighted integrals, which are 36.56 for the method without temporal loss, and 16.98 for the method with temporal loss. To verify that our temporal model actually establishes a stable temporal latent space instead of ignoring temporal information altogether, we evaluate the temporal frequencies for the same 100 inputs as above, but with a randomized order over time. In this case, our model correctly identifies the incoherent inputs, and yields similarly high frequencies as the regular model with 28.44 and 35.24, respectively. More details in Appendix C. In addition, we evaluated the changes of generated outputs over time w.r.t. ground truth motion. For this we mapped the generated point cloudsỸ t = {y t 1, y t 2, ..., y t n} for 100 frames to evenly and dense sampled ground-truth points on the original mesh Y t = {y t 1, y t 2, ..., y t n} (the moving man shown in Fig. 1). This gives us a dense correlation between the data and the generated point clouds. For the mapping we used an assignment based on nearest neighbors. γ:ỹ → y. Using γ we divideỸ t into n subsetsŶ i = {ỹ j |γ(ỹ j) = y i } which correlate with the corresponding ground-truth points. For each subset we can now compute the mean position 1 |Ŷi| ŷ∈Ŷiŷ and the sample density |Ŷ i | measured by the number of points assigned to a ground-truth sample position. The temporal change of these values are of particular interest. The change of the mean positions should correspond to the ground-truth changes, while the change of the density should be as small as possible. We have evaluated the error of the first and second derivative of positions, as well as the first and second derivative of density (see Fig. 9 and Table 3). As can be seen from the plots, our method leads to clear improvements for all measured quantities. The individual spikes that are visible for both versions in the position errors (c,d) most likely correspond to sudden changes of the input motions for which our networks undershoots by producing a smooth version of the motion. Table 3: Measurements averaged over 100 frames for a version of our network without temporal loss ("w/o") and with our full temporal loss formulation ("with"). The left table shows the for the error evaluation of the velocity and the acceleration, whereas in the right table one can see the variance of the density derivatives. Our patch-based approach currently relies on a decomposition of the input volumes into patches over time, as outlined in Appendix A. As all of the following involve temporal data, full sequences are provided in the accompanying video. We apply our method to several complex 3D models to illustrate its performance. Fig. 7 shows the input as well as several frames generated with our method for an animation of a spider. Our method produces an even and temporally stable reconstruction of the object. In comparison, Fig. 7b) shows the output from the previous work architecture. It exhibits uneven point distributions and outliers, e.g., above the legs of the spider, in addition to uneven motions. A second example for a moving human figure is shown in Fig. 1. In both examples, our network covers the target shape much more evenly despite using much fewer points, as shown in Table 2. Thanks to the flexible output size of our network, it can adapt to sparsely covered regions by generating correspondingly fewer outputs. The previous work architecture, with its fixed output size, needs to concentrate the fixed number of output points within the target shape, leading to an unnecessarily large point count. In order to demonstrate the flexibility of our method, we also apply it to a volumetric moving point cloud obtained from a liquid simulation. Thanks to the patch-based evaluation of our network, it is agnostic to the overall size of the input volume. In this way, it can be used to generate coherent sets with millions of points. These examples also highlight our method's capabilities for generalization. While the 3D model was only trained on data from physics simulations, as outlined above, it learns stable features that can be flexibly applied to volumetric as well as to surface-based data. The metrics in Table 1 show that for both 2D and 3D cases, our method leads to significantly improved quality, visible in lower loss values for spatial as well as temporal terms. Another interesting field of application for our algorithm are physical simulations. Complex simulations such as fluids, often employ particle-based representations. On the one hand, the volume data is much larger than surface-based data, which additionally motivates our dynamic output. On the other hand, time stability plays a very important role for physical phenomena. Our method produces detailed outputs for liquids, as can be seen in our supplemental video. Convergence graphs for the different versions are shown in Fig. 12 of the supplemental material. These graphs show that our method not only successfully leads to very low errors in terms of temporal coherence, but also improves spatial accuracy. The final values of L S for the 2D case are below 0.05 for our algorithm, compared to almost 0.08 for previous work. For 3D, our approach yields 0.04 on average, in contrast to ca. 0.1 for previous work. We have proposed a first method to infer temporally coherent features for point clouds. This is made possible by a combination of a novel loss function for temporal coherence in combination with enabling flexible truncation of the . In addition we have shown that it is crucial to prevent static patterns as easy-to-reach local minima for the network, which we avoid with the proposed a mingling loss term. Our super-resolution above demonstrate that our approach takes an important first step towards flexible deep learning methods for dynamic point clouds. Looking ahead, our method could also be flexibly combined with other network architectures or could be adopted for other applications. Specifically, a combination with PSGN could be used to generate point clouds from image sequences instead of single images. Other conceivable applications could employ methods like with our approach for generating animated meshes. Due to the growing popularity and ubiquity of scanning devices it will, e.g., be interesting to investigate classification tasks of 3D scans over time as future work. Apart from that, physical phenomena such as elastic bodies and fluids can likewise be represented in a Lagrangian manner, and pose interesting challenges and complex spatio-temporal changes. Data Generation We employ a physical simulation to generate our input and output pairs for training. This has the advantage that it leads a large variety of complex motions, and gives full control of the generation process. More specifically, we employ the IISPH algorithm, a form of Lagrangian fluid simulator that efficiently generates incompressible liquid volumes. These simulations also have the advantage that they inherently control the density of the point sampling thanks to their volume conserving properties. In order to generate input pairs for training, we randomly sample regions near the surface and extract points with a given radius around a central point. This represents the high-resolution target. To compute the low-resolution input, we downsample the points with a Poisson-disk sampling to compute a point set with the desired larger spacing. In order to prevent aliasing from features below the coarse resolution, we perform a pass of surface fairing and smoothing before downsampling. Due to the large number of patches that can be extracted from these simulations, we did not find it necessary to additionally augment the generated data sets. Examples of the low-and high-resolution pairs are shown in the supplemental material. Below we will demonstrate that models trained with this data can be flexibly applied to moving surface data as well as new liquid configurations. The surface data is generated from animated triangle meshes that were resampled with bicubic interpolation in order to match a chosen average per-point area. This pattern was generated once and then propagated over time with the animation. When applying our method to new liquid simulations, we do not perform any downsampling, but rather use all points of a low-resolution simulation directly, as a volumetric re-sampling over time is typically error prone, and gives incoherent low resolution inputs. Given a moving point cloud, we decompose it into temporally coherent patches in the following manner: We start by sampling points via a Poisson-disk sampling in a narrow band around the surface, e.g., based on a signed distance function computed from the input cloud. These points will persist as patch centers over time, unless they move too close to others, or too far away from the surface, which triggers their deletion. In addition, we perform several iterations for every new frame to sample new patches for points in the input cloud that are outside all existing patches. Note that this resampling of patches over time happens instantaneously in our implementation. While a temporal fading could easily be added, we have opted for employing transitions without fading, in order to show as much of the patch content as possible. Network Architecture and Training Our architecture heavily relies on a form of hierarchical point-based convolutions. I.e., the network extracts features for a subset of the points and their nearest neighbors. For the point convolution, we first select a given number of group centers that are evenly distributed points from a given input cloud. For each group center, we then search for a certain number of points within a chosen radius (a fraction of the [-1,1] range). This motivates our choice for a coordinate far outside the regular range for the padded points from Sec. 3.2. They are too far away from all groups by construction, so they are filtered out without any additional overhead. In this way, both feature extraction and grouping operations work flexibly with the varying input sizes. Each group is then processed by a PointNet-like sub-structure , yielding one feature vector per group. The is a set of feature vectors and the associated group position, which can be interpreted as a new point cloud to repeatedly apply a point convolution. In this way, the network extracts increasingly abstract and global features. The last set of features is then interpolated back to the original points of the input. Afterwards a sub-pixel convolution layer is used to scale up the point cloud extended with features and finally the final position vectors are generated with the help of two additional shared, fully connected layers. While we keep the core network architecture unmodified to allow for comparisons with previous work, an important distinction of our approach is the input and output masking, as described in Sec. 3.2. Our point data was generated with a mean point spacing, i.e., Poisson disk radius, of 0.5 units. For the 2D tests, an upscaling factor of r = 9 was used. For this purpose, patches with a diameter of 5 were extracted from the low-resolution data and patches with a diameter of 15 from the high-resolution data. We used the thresholds k max = 100 and n max = 900. For the loss, we used γ = 10, µ = 10, and ν = 0.001. The network was trained with 5 epochs for a data set with 185k pairs, and a batch size of 16, the learning rate was 0.001 with a decay of 0.003. For the 3D below, the scaling factor r was set to 8. The diameter of the patches was 6 for the low-resolution data and 12 for the high-resolution data, with k max = 1280 and n max = 10240. The loss parameters were γ = µ = 5, with ν = 0.001. Learning rate and decay were the same for training, but instead we used 10 epochs with 54k patches in 3D, and a batch size of 4. The input feature vector is processed in the first part of our network, which consists of four point convolutions. We use (n g, r g, [l 1, ..., l d]) to represent a level with n g groups of radius r g and [l 1, ..., l d] the d fully connected layers with the width l i (i = 1, ..., d). The parameters we use are (k max, 0.25, ), (k max /2, 0.5, ), (k max /4, 0.6, ) and (k max /8, 0.7, ). We then use interpolation layers to distribute the features of each convolution level among the input points. In this step, we reduce the output of each convolution layer with one shared, fully-connected layer per level, to a size of 64 and then distribute the features to all points of the input point cloud depending on their position. This extends the points of our original point cloud with 256 features. Fig. 11 shows a visual overview of the data flow in our network. Afterwards, we process the data in r separate branches consisting of two shared, fully interconnected layers with 256 and 128 nodes. The output is then processed with two shared fully connected layers of 64 and 3 nodes. Finally, we add our ing data to the input positions that have been repeated r times. This provides an additional skip connection which leads to slightly more stable . All convolution layers and fully interconnected layers use a tanh activation function. For the input feature vector, we make use of additional data fields in conjunction with the point positions. Our network also accepts additional features such as velocity, density and pressure of the SPH simulations used for data generation. For inputs from other sources, those values could be easily computed with suitable SPH interpolation kernels. In practice, we use position, velocity and pressure fields. Whereas the first two are important (as mentioned in Sec. 3.1), the pressure fields turned out to have negligible influence. In this section we give details for the frequency evaluation of Sec. 4. In order to measure the stability of the latent space against temporal changes, we evaluated the latent space of our network with and without temporal loss, once for 100 ordered patch sequences and once for 100 un-ordered ones. The central latent space of our network consists of the features generated by the point-convolution layers in the first part of the network and is 256 dimensional (see Fig. 11). To obtain information about its general behavior, we average the latent space components over all 100 patch sequences, subtract the mean, and normalize the ing vector w.r.t. maximum value for each data set. The is a time sequence of scalar values representing the mean deviations of the latent space. The Fourier transform of these vectorsf, are shown in Fig. 8, and were used to compute the weighted frequency content Figure 10: Examples from our synthetic data generation process. In (a,b) each a high resolution reference frame is shown in purple, and in green the down-sampled low resolution frames generated from it. The training data is generated by sampled patches from these volumes. Figure 11: An overview of our network architecture. The first row shows the hierarchical point convolutions, while the bottom rows illustrate the processing of extracted features until the final output point coordinates are generated. Figure 12: Convergence plots for the training runs of our different 2D and 3D versions. The combined loss only illustrates convergence behavior for each method separately, as weights and terms differ across the four variants. LM for previous work is not minimized, and only given for reference."

iSparse eliminates irrelevant or insignificant network edges with minimal impact on network performance by determining edge importance w.r.t. the final network output.

Represent the short summary for finding the corresponding scientific article (long)


Question: represent the article for finding a single-sentence tldr!

Question: Language modeling tasks, in which words are predicted on the basis of a local context, have been very effective for learning word embeddings and context dependent representations of phrases. Motivated by the observation that efforts to code world knowledge into machine readable knowledge bases tend to be entity-centric, we investigate the use of a fill-in-the-blank task to learn context independent representations of entities from the contexts in which those entities were mentioned. We show that large scale training of neural models allows us to learn extremely high fidelity entity typing information, which we demonstrate with few-shot reconstruction of Wikipedia categories. Our learning approach is powerful enough to encode specialized topics such as Giro d’Italia cyclists. A long term goal of artificial intelligence has been the development and population of an entitycentric representation of human knowledge. Efforts have been made to create the knowledge representation with knowledge engineers BID10 or crowdsourcers BID1. However, these methods have relied heavily on human definitions of their ontologies, which are both limited in scope and brittle in nature. Conversely, due to recent advances in deep learning, we can now learn robust general purpose representations of words BID13 and contextualized phrases BID16 BID6 directly from large textual corpora. Consider the following context in which an entity mention is replaced with the [MASK] symbol:... [MASK], a Russian factory worker, was the first woman in space...As readers, we understand that first woman in space is a unique identifier, and we are able to fill in the blank unambiguously. The central hypothesis of this paper is that, by matching entities to the contexts in which they are mentioned, we should be able to build a representation for Valentina Tereshkova that encodes the fact that she was the first woman in space. To do this, we start with BERT BID6, a powerful pretrained text encoder, to encode contexts-Wikipedia text in which a hyperlinked span has been blanked out-and we train an entity encoder to match the BERT representation of the entity's contexts. We experiment with a lookup table that maps each entity to a fixed length vector, which we call RELIC (Representations of Entities Learned In Context). We hypothesize that the dedicated entity representations in RELIC should be able to capture knowledge that is not present in BERT. To test this, we compare RELIC to two BERT-based entity encoders: one that encodes the entity's canonical name, and one that encodes the first paragraph of the entity's Wikipedia page. Ultimately, we would like our representations to encode all of the salient information about each entity. However, for this initial work, we study our representations' ability to capture Wikipedia categorical information encoded by human experts. We show that given just a few exemplar entities of a Wikipedia category such as Giro d'Italia cyclists, we can use RELIC to recover the remaining entities of that category with good precision. Several works have tackled the problem of learning distributed representations of entities in a knowledge base (KB). Typical approaches rely on the (subject, relation, object) ontology of KBs like Free-base BID1. These methods embed the entities and relations in vector space, then maximize the score of observed triples against negative triples to do KB completion BID3 BID21 BID28 BID24.There is relatively less work in learning entity representations directly from text. The contextual approach of word2vec BID13 has been applied to entities, but there has been little analysis of how effective such a method would be for answering the entity typing queries we study in this work. Most methods for entity representations that do use raw text will combine it with structure from an existing KB BID18 BID23 BID9 in an effort to leverage as much information as possible. While there may be gains to be had from using structure, our goal in this work is to isolate and understand the limits of inducing entity representations from raw text alone. We also note the similarity of our RELIC task to entity linking BID17 BID4 BID22 BID7 and entity typing BID27 BID19, where entity mentions are processed in context. Unlike previous work in context-dependent entity typing BID11 BID5 BID15, we consider types of RELIC from a global perspective. We are interested in identifying contextindependent types of entities so that they can be used to identify structure in the entity latent space. We study the ability of current models to learn entity encoders directly from the contexts in which those entities are seen. Formally, we define an entity encoder to be a function h e = f (e) that maps each entity e to a vector h e ∈ R d. We outline the training procedure used to learn the encoders. RELIC training input Let E = {e 0 . . . e N} be a predefined set of entities, and let DISPLAYFORM0 is a sequence of words x i ∈ V. Each context contains exactly one instance of the [MASK] symbol. Our training data is a corpus of (context, entity) pairs DISPLAYFORM1. Each y i ∈ E is an entity, and the [MASK] symbol in x i substitutes for a single mention y i. For clean training data, we extract our corpus from English Wikipedia, taking advantage of its rich hyperlink structure (Section 3.2). We introduce a context encoder h x = g(x) that maps the context x into the same space R d as our entity encodings. Then we define a compatibility score between the entity e and the context x as the scaled cosine similarity s(x, e) = a · DISPLAYFORM0 ||g(x)||||f (e)|| where the scaling factor a is a learned parameter, following BID26. Now, given a context x, the conditional probability that e was the entity seen with x is defined as p(e|x) = exp(s(x,e)) e ∈E exp(s(x,e)) and we train RELIC by maximizing the average log probability 1 |D| (x,y)∈D log p(y|x). In practice, we use a noise contrastive loss BID8 BID14 ), where we sample K negative entities e 1, e 2,..., e K from a noise distribution p noise (e). Denoting e 0:= e, our per-example loss is l(s, x, e) = − log DISPLAYFORM1. We train our model with minibatch gradient descent and use all other entries in the batch as negatives. This is roughly equivalent to p noise (e) being proportional to entity frequency. BERT context encoder For g, we use the pretrained BERT model BID6, a powerful Transformer-based BID25 text encoder, to encode contexts into a fixed-size representation 1. We project the BERT hidden state into R d using a weight matrix W ∈ R d×768 to obtain our context encoding. Table 1: Results for the Wikipedia category population task. Mean Average Precision for ranking entities given a set of exemplars of a given Wikipedia class. K represents the number of candidates to be ranked, andp is the average number of positive labels among the candidates. Results are averaged over 100 categories sampled at random from those containing at least 1000 entities. Wikipedia name, and the first paragraph of its Wikipedia page. We consider three encoders that operate on different representations of the entities: embedding lookup, BERT name encoder, and BERT description encoder. In the standard RELIC setup, we map each entity, identified by its unique ID, directly onto its own dedicated vector in R d via a |E| × d dimensional embedding matrix. We also consider two alternate BERT-based options for distributed encoding of entities, which are fine-tuned on the RELIC data. The name encoder applies a BERT Transformer to the canonical name of the entity to obtain a fixedsize representation. The description encoder applies a BERT Transformer to an entity's description to obtain a fixed size representation 3. Note that both name and description encoders can do zero-shot encoding of new entities, assuming that a name or description is provided. To train RELIC, we obtain data from the 2018-10-22 dump of English Wikipedia. We take E to be the set of all entities in Wikipedia (of which there are over 5 million). For each hyperlink, we take the context as the surrounding sentence, replace all tokens in the anchor text with a single [MASK] symbol, and set the entity linked to as ground truth. We limit each context to 64 tokens. We set the entity embedding size to d = 300. For the name and description encoders, we take the initial hidden state of the Transformer as the fixed-size representation. We limit to 16 name tokens and 128 description tokens. We train the model using TensorFlow BID0 ) with a batch size of 1024 for 5 epochs. We hold out about 1/30,000 of the data for use as validation, on which the final model achieves roughly 85% accuracy in-batch negative prediction for all models. We introduce a fine-grained entity typing task based on Wikipedia categories, where the task is to populate a category from a small number of exemplars. We evaluate if RELIC benefits from dedicated embeddings over the BERT encoders that share parameters between entities. We filter the Wikipedia categories in Yago 3.1 BID12 to get the 1,129 categories that cover at least 1,000 entities and consider an exemplar based "few-shot" scenario, based on the prototypical approach of BID20. For each category, we provide a small number of exemplars (3 or 10), one correct candidate entity drawn from the category, and K −1 other candidate entities. The candidate entities are ranked according to the inner product between their RELIC embeddings and the centroid of the exemplar embeddings, and we report the mean average precision (MAP) for entities belonging to the query class. Wikipedia categories are often incompletely labeled, and when K covers all entities, this confounds the MAP calculation. Therefore, we also present for K = 10 and K = 1000 for a cleaner experimental setup. TAB2 show . RELIC outperforms both the BERT name and description encoders when we restrict the candidate set to the entities seen at least 10 times in RELIC's training data, and the gap in performance increases as we increase the entity frequency threshold. However, both the name and description encoders outperform RELIC on very infrequent entities, since they can generalize from other entities with similar naming conventions or descriptions, while RELIC's embedding matrix treats every entity completely separately. FIG0 shows examples of predictions for Wikipedia categories given 3 exemplars for 5 randomly sampled categories. Most categories show high precision in the top 10 predictions. The category Butterflies of Africa fails-this is likely due to the fact that the 3 exemplars appeared only a total of 4 times in our pretraining data. The Giro d'Italia cyclists category is very well predicted-the single incorrect prediction Thibaut Pinot did cycle in the Giro d'Italia. However, for Video games featuring female protagonists, most of RELIC's success is due to just retrieving variations of the Final Fantasy series. We demonstrated that the RELIC fill-in-the-blank task allows us to learn highly interesting representations of entities with their own latent ontology, which we empirically verify through a few-shot Wikipedia category reconstruction task. We encourage researchers to explore the properties of our entity representations and BERT context encoder, which we will release publicly.

Represent the short summary for finding the corresponding scientific article (long)

We learn entity representations that can reconstruct Wikipedia categories with just a few exemplars.

Represent the TLDR for finding the corresponding scientific article

Spoken Term Detection, using structured prediction and deep networks, implementing a new loss function that both maximizes AUC and ranks according to a predefined threshold.


Question: Represent the article for finding a single-sentence TLDR

Question: Variational autoencoders (VAEs) have been successful at learning a low-dimensional manifold from high-dimensional data with complex dependencies. At their core, they consist of a powerful Bayesian probabilistic inference model, to capture the salient features of the data. In training, they exploit the power of variational inference, by optimizing a lower bound on the model evidence. The latent representation and the performance of VAEs are heavily influenced by the type of bound used as a cost function. Significant research work has been carried out into the development of tighter bounds than the original ELBO, to more accurately approximate the true log-likelihood. By leveraging the q-deformed logarithm in the traditional lower bounds, ELBO and IWAE, and the upper bound CUBO, we bring contributions to this direction of research. In this proof-of-concept study, we explore different ways of creating these q-deformed bounds that are tighter than the classical ones and we show improvements in the performance of such VAEs on the binarized MNIST dataset. Variational autoencoders (VAEs) BID10, BID4 ) are powerful Bayesian probabilistic models, which combine the advantages of neural networks with those of Bayesian inference. They consist of an encoder created with a neural network architecture, which maps the high-dimensional input data, x, to a low-dimensional latent representation, z, through the posterior probability distribution, p(z|x). Then, samples from this latent distribution are decoded back to a high-dimensional signal, through another neural network architecture and the probability distribution p(x|z). Integration performed with these probability distributions from the Bayesian framework of VAEs is intractable. As a solution, variational inference is employed to perform learning in these models, whereby a tractable bound on the model evidence is optimized instead of the intractable model evidence itself BID3. By design, the output model is set as p(x|z), usually a Bernoulli or a Gaussian probability distribution, depending on whether the target is discrete or continuous, and the prior distribution of the latent space as p(z). However, the true posterior distribution, p(z|x), remains unknown and is intractable. To solve this issue, an approximate posterior distribution, q(z|x), is learnt by means of a lower bound on the model evidence, termed the ELBO. For one data point, x (i), writing out the Kullback-Leibler divergence between the true and approximate posterior distributions and using its positivity property yields this bound: DISPLAYFORM0 The lower bound on the model evidence, the ELBO, now becomes the cost function used during the training phase of the VAEs. Over time, the first term shows how the reconstruction loss changes and the second term how far the approximate posterior is to the prior distribution. The of inference and the performance of VAEs on reconstructing and generating images heavily depend on the type of bound employed in training. A significant body of work has been carried out to replace the ELBO with tighter bounds on the model evidence. On the one hand, starting from an unbiased estimator of the true log-likelihood, the authors of BID0 derive an importance sampling estimate of the model evidence, the IWAE. This represents one of the tightest bounds of VAEs and has only recently been improved on in BID8, BID11. Increasing the number of importance samples in the IWAE objective, decreases the signal-to-noise-ratio of the gradients, which makes the learning more difficult, as the gradients suffer from a larger level of noise BID8. Several strategies are able to correct this issue. In the first algorithm, MIWAE, the outer expectation of the IWAE objective is approximated with more than one sample, as is the case in the IWAE. The second algorithm, CIWAE, represents a convex combination of the ELBO and the IWAE bounds and the third algorithm, PIWAE, separately trains the encoder and the decoder networks with different IWAE objectives. On the other hand, leveraging different divergences between the true and the approximate posterior distributions has lead to diverse bounds on the model evidence. Starting from the Rényi α-divergence BID9 between such distributions, a family of lower and upper bounds are obtained, parameterized by α BID6. However, these lower bounds become competitive with the IWAE, only in the limit α → −∞. In addition, the upper bounds suffer from approximation errors and bias and the means to select the best value of the hyperparameter α is unknown. Through an importance sampling scheme similar to the one found in the IWAE, these Rényi α bounds are tightened in BID15. If the Rényi α-divergence is replaced with the χ 2 divergence, the bound on the model evidence becomes the upper bound CUBO BID1. The Rényi α-family of bounds and others lose their interpretability as a reconstruction loss and a Kullback-Leibler divergence term that measures how close the approximate posterior is to the prior distribution. They remain just a cost function optimized during training. With different compositions of convex and concave functions, the approaches described above are unified in the K-sample generalized evidence lower bound, GLBO BID11. This study generalizes the concept of maximizing the logarithm of the model evidence to maximizing the φ-evidence score, where φ(u) is a concave function that replaces the logarithm. It allows for great flexibility in the choice of training objectives in VAEs. One particular setting provides a lower bound, the CLBO, which surpasses the IWAE objective. The aim of this work is to leverage the theory of q-deformed functions introduced in BID12, BID13, BID14, to derive tighter lower bounds on the model evidence in VAEs. To this end, our contributions are three-fold: firstly, we derive two novel lower bounds, by replacing the logarithm function in the classical ELBO, BID10, BID4, and IWAE bounds, BID0, BID7, respectively, with the q-deformed logarithm function. Values of q < 1.0 yield upper bounds of varying tightness on the classical logarithm function, as illustrated in FIG0.Secondly, we combine the information given by the upper bound CUBO, BID1, with the information given by the ELBO and the IWAE, respectively, to obtain a lower bound that is placed between the two. By the means of their construction, we hypothesize these q-deformed bounds to be closer to the true log-likelihood. We are able to confirm it in our experiments. We term our novel lower bounds the qELBO and the qIWAE.Thirdly, the tighteness of the gap between the classical logarithm function and the q-deformed one depends on the value of q, as seen in FIG0. Thus, q becomes a hyperparameter of our algorithm. Since q is a number, we can optimize it efficiently and accurately, using standard optimization algorithms. By solving for the best q for each data batch, we make q a data-driven hyperparameter, tuned in an adaptive way during training. With the q-entropy, introduced in BID12, the author developed the field of nonextensive statistical mechanics, as a generalization of traditional statistical mechanics, centered around the Boltzmann-Gibbs distribution. The S q entropy provides a generalization of this distribution, which can more accurately explain the phenomena of anomalous physical systems, characterized by rare events. In the following definitions, the original quantities can be recovered in the limit q → 1. If k > 0 is a constant, W ∈ N is the total number of possible states of a system and p i the corresponding probabilities, ∀i = 1: W, then: DISPLAYFORM0 The generalized logarithmic function, termed the q-logarithm, is introduced in BID13 as: DISPLAYFORM1 The Kullback-Leibler divergence is generalized in BID14 to the form DISPLAYFORM2 In order to derive our q-deformed bounds, we replace the logarithm function from the ELBO and IWAE bounds, with its q-deformed version. By appropriately optimizing the hyperparameter q, we will obtain an upper bound on the ELBO and IWAE, respectively: DISPLAYFORM3 DISPLAYFORM4 Optimization algorithm for q. We train a variational autoencoder with our novel qELBO and qIWAE bounds. The training procedure and the optimization method for q are identical for both types of q-deformed bounds. We will describe them in the case of the qELBO.We start the training procedure with an initial value of q = 1.0 − 10 −6. For one batch of images, we compute the qELBO lower bound and the CUBO upper bound BID1, averaged over the batch. In order to obtain a tighter lower bound, qELBO *, we set a desired value of the cost function at qELBO * = qELBO +τ · (CUBO − qELBO), where, in our experiments, τ ∈ {0.5, 0.75}.By means of the L-BFGS-B optimization method, we find the optimal value q *, such that DISPLAYFORM5 For this task, we employ the scipy optimization package in python. We apply the gradient descent step on our new, improved, cost function, qELBO *, computed with this optimal value, q *. We save this value of q for the next batch of images and we repeat the optimization steps described above, for all training batches. For the experiments conducted on the MNIST dataset BID5, we use the one-stochastic layer architecture employed in BID0 and in BID6. The encoder and the decoder are composed of two deterministic layers, each with 200 nodes, and of a stochastic layer with 50 nodes. The dimension of the latent space is equal to 50 and the activation functions are the softplus function. The approximate posterior is modeled as a Gaussian distribution, with a diagonal covariance matrix. The output model is a Bernoulli distribution for each pixel. We use the binarized MNIST dataset provided by tensorflow, with 55000 training images and 10000 test images. The learning rate is fixed at 0.005 and there is no updating schedule. To implement and test our new algorithms, we modify publicly available code 1 BID6. On the benchmark binary MNIST dataset BID5, we compare our newly derived q-deformed bounds with the ELBO and the IWAE and we show several improvements that we obtained. On the test set, we report the bounds computed with K number of samples and the true log-likelihood estimated with 5000 importance samples, logp x. The expectations involved in all of the bounds are estimated with Monte Carlo sampling. For the ELBO and the qELBO bounds, the expectation is approximated with K number of samples. The expectation in the standard IWAE is approximated with one sample. Thus, we will compute the expectation in the qIWAE with one sample, as well. Here, K refers to the number of importance samples used in the computation of the bound. In addition, we illustrate the performance of our algorithms on reconstructed binary MNIST test images and on randomly generated ones. After 3000 epochs of training, the qIWAE(τ =0.5) Figure 3: Method: qVAE(τ = 0.5) with K=50 samples. From left to right: original binary MNIST test images, reconstructed and randomly generated ones.algorithm, with the bound estimated with K=50 samples, gives the best on the importance sampling estimate of the true log-likelihood, very close to the one given by the standard IWAE. Moreover, the q-deformed bound is much closer to the estimated true value, than is the IWAE bound. We observe this behaviour for all the q-deformed bounds. This implies that, during training, optimizing the q-deformed bounds provides a cost function that is a more accurate approximation of the model evidence. Although the q-deformed ELBO does not outperform the standard IWAE, we can see significant improvements over the traditional ELBO, in all the test cases. A large decrease in the value of the bound is present for all the qELBO variants, more pronounced in the large sample regime. We addressed the challenging task of deriving tighter bounds on the model evidence of VAEs. Significant research effort has gone in this direction, with several major contributions having been developed so far, which we reviewed in the introduction. We leveraged the q-deformed logarithm function, to explore other ways of tightening the lower bounds. As well as improvements in the estimated true log-likelihood, we found that the q-deformed bounds are much closer to the estimated true log-likelihood, than the classical bounds are. Thus, training with our novel bounds as the cost function may increase the learning ability of VAEs. Through the preliminary experiments we have conducted so far, we have achieved our goal. They show that our approach has merit and that this direction of research is worth pursuing in more depth, to produce more accurate bounds and to study their impact on the performance of VAEs. As future work, similarly to BID8, we plan to investigate how the tightening the ELBO and the IWAE influences the learning process and affects the gradients and the structure of the latent space, compared with the classical case. In addition, we plan to explore different optimization strategies for q and to study its role in achieving tighter bounds. We will also apply our q-deformed bounds, to investigate the disentanglement problem in VAEs, see for example BID2. The research question addressed here is how different bounds change the structure of the latent space, to provide better or worse disentanglement scores. Finally, we would also like to test our novel bounds on all the major benchmark datasets used for assessing the performance of VAEs and compare them with other state-of-the-art bounds on the model evidence.

represent the tldr for finding the corresponding scientific article

Using the q-deformed logarithm, we derive tighter bounds than IWAE, to train variational autoencoders.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.