Question: represent the text to find a very short summary (1-2 sentences max)

Question: Point clouds, as a form of Lagrangian representation, allow for powerful and flexible applications in a large number of computational disciplines. We propose a novel deep-learning method to learn stable and temporally coherent feature spaces for points clouds that change over time. We identify a set of inherent problems with these approaches: without knowledge of the time dimension, the inferred solutions can exhibit strong flickering, and easy solutions to suppress this flickering can in undesirable local minima that manifest themselves as halo structures. We propose a novel temporal loss function that takes into account higher time derivatives of the point positions, and encourages mingling, i.e., to prevent the aforementioned halos. We combine these techniques in a super-resolution method with a truncation approach to flexibly adapt the size of the generated positions. We show that our method works for large, deforming point sets from different sources to demonstrate the flexibility of our approach. Deep learning methods have proven themselves as powerful computational tools in many disciplines, and within it a topic of strongly growing interest is deep learning for point-based data sets. These Lagrangian representations are challenging for learning methods due to their unordered nature, but are highly useful in a variety of settings from geometry processing and 3D scanning to physical simulations, and since the seminal work of , a range of powerful inference tasks can be achieved based on point sets. Despite their success, interestingly, no works so far have taken into account time. Our world, and the objects within it, naturally move and change over time, and as such it is crucial for flexible point-based inference to take the time dimension into account. In this context, we propose a method to learn temporally stable representations for point-based data sets, and demonstrate its usefulness in the context of super-resolution. An inherent difficulty of point-based data is their lack of ordering, which makes operations such as convolutions, which are easy to perform for Eulerian data, unexpectedly difficult. Several powerful approaches for point-based convolutions have been proposed (; ;), and we leverage similar neural network architectures in conjunction with the permutation-invariant Earth Mover's Distance (EMD) to propose a first formulation of a loss for temporal coherence. In addition, several works have recognized the importance of training point networks for localized patches, in order to avoid having the network to rely on a full view of the whole data-set for tasks that are inherently local, such as normal estimation , and super-resolution ). This also makes it possible to flexibly process inputs of any size. Later on we will demonstrate the importance of such a patch-based approach with sets of changing cardinality in our setting. A general challenge here is to deal with varying input sizes, and for super-resolution tasks, also varying output sizes. Thus, in summary we target an extremely challenging learning problem: we are facing permutation-invariant inputs and targets of varying size, that dynamically move and deform over time. In order to enable deep learning approaches in this context, we make the following key contributions: Permutation invariant loss terms for temporally coherent point set generation; A Siamese training setup and generator architecture for point-based super-resolution with neural networks; Enabling improved output variance by allowing for dynamic adjustments of the output size; The identification of a specialized form of mode collapse for temporal point networks, together with a loss term to remove them. We demonstrate that these contributions together make it possible to infer stable solutions for dynamically moving point clouds with millions of points. More formally, we show that our learning approach can be used for generating a point set with an increased resolution from a given set of input points. The generated points should provide an improved discretization of the underlying ground truth shape represented by the initial set of points. For the increase, we will target a factor of two to three per spatial dimension. Thus, the network has the task to estimate the underlying shape, and to generate suitable sampling positions as output. This is generally difficult due to the lack of connectivity and ordering, and in our case, positions that move over time in combination with a changing number of input points. Hence it is crucial that the network is able to establish a temporally stable latent space representation. Although we assume that we know correspondences over time, i.e., we know which point at time t moved to a new location at time t + ∆t, the points can arbitrarily change their relative position and density over the course of their movement, leading to a substantially more difficult inference problem than for the static case. Deep learning with static point sets was first targeted in PointNet via order-invariant networks, while PointNet++ (extended this concept to generate features for localized groups similar to a convolution operation for grid-based data. This concept can be hierarchically applied to the generated groups, in order to extract increasingly abstract and global features. Afterwards, the extracted features can be interpolated back to the original point cloud. The goal to define point convolutions has been explored and extended in several works. The MCNN approach phrased convolution in terms of a Monte Carlo integration. PointCNN defined a pointwise convolution operator using nearest neighbors, while extension-restriction operators for mapping between a point cloud function and a volumetric function were used in. The PU-Net proposed a network for upsampling point clouds, and proposed a similar hierarchical network structure of PointNets along the lines of PointNet++ to define convolutions. Being closer to our goals, we employ this approach for convolutional operations in our networks below. We do not employ the edge-aware variant of the PU-Net (b) here, as we focus on temporal changes in our work. Permutation invariance is a central topic for point data, and was likewise targeted in other works . The Deep Kd-network defined a hierarchical convolution on point clouds via kd-trees. PointProNets employed deep learning to generate dense sets of points from sparse and noisy input points for 3D reconstruction applications. PCPNet , as another multi-scale variant of PointNet, has demonstrated high accuracy for estimating local shape properties such as normal or curvature. P2PNet ) used a bidirectional network and extends PointNet++ to learn a transformation between two point clouds with the same cardinality. Recently, the area of point-based learning has seen a huge rise in interest. One focus here are 3D segmentation problems, where numerous improvements were proposed, e.g., by SPLATNet , SGPN (a), SpiderCNN , PointConv , SONEt and 3DRNN . Other networks such as Flex Convolution , the SuperPoint Graph , and the fully convolutional network focused on large scale segmentation. Additional areas of interest are shape classification (b; ; ;) and object detection , and hand pose tracking . Additional works have targeted rotation and translation invariant inference , and point cloud autoencoders ). A few works have also targeted generative models based on points, e.g., for point cloud generation, and with adversarial approaches. It is worth noting here that despite the huge interest, the works above do not take into account temporally changing data, which is the focus of our work. A notable exception is an approach for scene flow, in order to estimate 3D motion directly on the basis of point clouds. This work is largely orthogonal to ours, as it does not target generative point-based models. We assume an input point cloud, where d includes 3 spatial coordinates and optionally additional features. Our goal is to let the network f s (X) infer a functionỸ which approximates a desired super-resolution output point cloud Y = {y 1, y 2, ..., y n} of size n ∈ [1, n max] with y i ∈ R 3, i.e. f s (X) =Ỹ ≈ Y. For now we assume that the number of output points n is defined by multiplying k with a user-defined upsampling factor r, i.e. n = rk. Figure 2a) illustrates the data flow in our super-resolution network schematically. We treat the upsampling problem as local one, i.e., we assume that the inference problem can be solved based on a spatially constrained neighborhood. This allows us to work with individual patches extracted from input point clouds. At the same time, it makes it possible to upsample adaptively, for example, by limiting the inference to relevant areas, such as complex surface structures. For the patch extraction we use a fixed spatial radius and normalize point coordinates within each patch to lie in the range of [−1, 1]. Our first building block is a measure for how well two point clouds represent the same object or scene by formulating an adequate spatial loss function. , we base our spatial loss L S on the Earth Mover's Distance (EMD), which solves an assignment problem to obtain a differentiable bijective mapping φ:ỹ → y. With φ we can minimize differences in position for arbitrary orderings of the points clouds via: Figure 2: a) Schematic overview of fs(X). Black arrows represent scalar data. Point data is depicted as colored arrows with the color indicating data cardinality (brown=k, red = kmax, green = nmax, blue = n, and purple =ñ). b) Siamese network setup for temporal loss calculation. When not taking temporal coherence explicitly into account, the highly nonlinear and ill-posed nature of the super-resolution problem can cause strong variations in the output even for very subtle changes in the input. This in significant temporal artifacts that manifest themselves as flickering. In order to stabilize the output while at the same time keeping the network structure as small and simple as possible, we propose the following training setup. Given a sequence of high resolution point clouds Y t, with t indicating time, we can compute a velocity For this we use a finite difference (y t+1 i − y t i), where we assume, without loss of generality, ∆t = 1, i.e. the time step is normalized to one. For training, the low resolution inputs X can now be generated from Y via down-sampling by a factor of r, which yields a subset of points with velocities. Details of our data generation process will be given below. To train a temporally coherent network with the Y t sequences, we employ a Siamese setup shown in Figure 2b. We evaluate the network several times (3 times in practice) with the same set of weights, and moving inputs, in order to enforce the output to behave consistently. In this way we avoid recurrent architectures that would have to process the high resolution outputs multiple times. In addition, we can compute temporal derivatives from the input points, and use them to control the behavior of the generated output. Under the assumption of slowly moving inputs, which theoretically could be ensured for training, a straightforward way to enforce temporal coherence would be to minimize the generated positions over consecutive time steps in terms of an L 2 norm: While this reduces flickering, it does not constrain the change of velocities, i.e., the acceleration. This in a high frequency jittering of the generated point positions. The jitter can be reduced by also including the previous state at time step t − 1 to constrain the acceleration in terms of its L 2 norm: However, a central problem of a direct temporal constraint via Equations and is that it consistently leads to a highly undesirable clustering of generated points around the center point. This is caused by the fact that the training procedure as described so far is unbalanced, as it only encourages minimizing changes. The network cannot learn to reconstruct realistic, larger motions in this way, but rather. can trivially minimize the loss by contracting all outputs to a single point. For this reason, we instead use the estimated velocity of the ground truth point cloud sequence with a forward difference in time, to provide the network with a reference. By using the EMD-based mapping φ established for the spatial loss in Equation, we can formulate the temporal constraint in a permutation invariant manner as Intuitively, this means the generated outputs should mimic the motion of the closest ground truth points. As detailed for the L 2 -based approaches above, it makes sense to also take the ground truth acceleration into account to minimize rapid changes of velocity over time. We can likewise formulate this in a permutation invariant way w.r.t. ground truth points via: We found that a combination of L EV and L EA together with the spatial loss L S from Eq. 1 provides the best , as we will demonstrate below. First, we will introduce the additional loss terms of our algorithm. Existing network architectures are typically designed for processing a fixed amount of input and output points. However, in many cases, and especially for a localized inference of super-resolution, the number of input and output points varies significantly. While we can safely assume that no patch exceeds the maximal number of inputs k max (this can be ensured by working on a subset), it can easily happen that a certain spatial region has fewer points. Simply including more distant points could guarantee that we have a full set of samples, but this would mean the network has to be invariant to scaling, and to produce features at different spatial scales. Instead, we train our network for a fixed spatial size, and ensure that it can process varying numbers of inputs. For inputs with fewer than k max points, we pad the input vector to have a fixed size. Here, we ensure that the padding values are not misinterpreted by the network as being point data. Therefore, we pad X with p ∈ {−2} d, which represents a value outside the regular patch coordinate range [−1, 1]: The first convolutional layer in our network now filters out the padded entries using the following mask: The entries of p allow us to compute the mask on the fly throughout the whole network, without having to pass through k. For an input of size k, our network has the task to generateñ = rk points. As the size of the network output is constant with rk max, the outputs are likewise masked with M out to truncate it to lengthñ for all loss calculations, e.g., the EMD mappings. Thus, as shown in Figure 2a,ñ is used to truncate the point cloudȲ = {ȳ 1,ȳ 2, ...,ȳ nmax} via a mask M out to form the final output Note that in Sec. 3.1, we have for simplicity assumed that n = rk, however, in practice the number of ground truth points n varies. As such,ñ only provides an approximation of the true number of target points in the ground truth data. While the approximation is accurate for planar surfaces and volumes, it is less accurate in the presence of detailed surface structures that are smaller than the spatial frequency of the low-resolution data. We have analyzed the effect of this approximation in Fig. 3. The histograms show that the strongly varying output counts are an important factor in practice, and Fig. 4 additionally shows the improvement in terms of target shape that from incorporating variable output sizes. In general,ñ provides a good approximation for our data sets. However, as there is a chance to infer an improved estimate of the correct output size based on the input points, we have experimented with training a second network to predictñ in conjunction with a differentiable M out. While this could be an interesting feature for future applications, we have not found it to significantly improve . As such, the evaluations and below will use the analytic calculation, i.e.,ñ = rk. For each input point the network generates r output points, which can be seen as individual groups g: ψ(g) = {Ỹ i |i ∈ [rg + 1, (r + 1)g]}. These groups of size r in the output are strongly related to the input points they originate from. Networks that focus on maintaining temporal coherence for the dynamically changing output tends to slide into local minima where r output points are attached as a L2 Loss Velocity Only Velocity + Acceleration Figure 6: Ablation study for our temporal loss formulation. Black points indicate targets, while green points are generated (both shown as time average). a) Result from previous work; b) With L2V loss; c) the proposed velocity loss LEV; d) our full loss formulation with LEV + LEA. While (a) has difficulties to approximate the target shape and the flickering output is visible as blurred positions, the additional loss terms (esp. in (c) and (d) Table 1: Quantitative for the different terms of our loss functions, first for our 2D ablation study and then for our 3D versions. The first three columns contain spatial, the next four temporal metrics. LN = ñ − n 2 2 is given as a measure of accuracy in terms of the size of the generated outputs (it is not part of the training). fixed structure to the input point location. This manifests itself as visible halo-like structures that move statically with the input. Although temporal coherence is good in this case, these cluster-like structures lead to gaps and suboptimal point distributions in the output, particularly over time. These structures can be seen as a form of temporal mode collapse that can be observed in other areas deep learning, such as GANs. To counteract this effect, we introduce an additional mingling loss term to prevent the formation of clusters by pushing the individual points of a group apart: Figure 5: Left, a without the mingling loss from Eq. 6, right with (a single point group highlighted in orange). The former has many repeated copies of a single pattern, which the mingling manages to distribute. Note that in contrast to previously used repulsion losses, L M encourages points to globally mix rather than just locally repelling each other. While a repulsion term can lead to a deterioration of the generated outputs, our formulation preserves spatial structure and temporal coherence while leading to well distributed points, as is illustrated in Fig. 5. In combination with the spatial and temporal terms from above, this leads to our final loss function L f inal = L S + γL EV + µL EA + νL M, with weighting terms γ, µ, ν. We train our network in a fully supervised manner with simulated data. To illustrate the effect of our temporal loss functions, we employ it in conjunction with established network architectures from previous work (; . Details of the data generation and network architectures are given in the appendix. We first discuss our data generation and training setup, then illustrate the effects of the different terms of our loss function, before showing for more complex 3D data sets. As our focus on temporal coherence, which is best seen a) b) c) Figure 7: Our method applied to an animation of a moving spider. (a) Input point cloud, (b) three frames of our method, (c) a detail from previous work (top) and our method (bottom). Note that our method at the bottom preserves the shape with fewer outliers, and leads to a more even distribution of points, despite generating fewer points in total (see Table 2). a) shows averaged latent space values for 100 random patch sequences of our 2D data set. The green curve shows our method with temporal coherence loss, while the pink curve was generated without it. The same data is shown in frequency space in (b), where the red curve represents the frequency of the data with temporal loss, and the blue curve the frequency of the data without. These graph highlights the reduced amount of high frequency changes in the latent space with temporal loss, esp. in frequency space, where the red curve almost entirely lies below the blue one. (c) contains frequency information for the latent space content of the same 100 patch sequences, but with a random order. In this case, the blue and red curve both contain significant amounts of high-frequencies. I.e., our method reliably identifies strongly changing inputs. Table 2: Point counts for the 3D examples of our video. Input counts together with output counts for previous work (P.W.) and our proposed network are shown. Factor columns contain increase in point set size from in-to output. As previous work cannot handle flexible output counts, a fixed number of points is generated per patch, leading to a huge number of redundant points. However, our network flexibly adapts the output size and leads to a significantly smaller number of generated points that cover the object or volume more evenly. in motion, we refer readers to the supplemental video at https://www.dropbox.com/sh/ btrzxavn34qftfe/AAADpIBME0eguA4ew4ylvCX_a?dl=0 in order to fully evaluate the ing quality. Ablation Study We evaluate the effectiveness of our loss formulation with a two dimensional ablation study. An exemplary patch of this study is shown in Fig. 6. In order to compare our method to previous work, we have trained a previously proposed method for point-cloud super-resolution, the PU-Net ) which internally uses a PointNet++ , with our data set, the only difference being that we use zero-padding here. This architecture will be used in the following comparisons to previous work. Fig. 6a ) shows a generated with this network. As this figure contains an average of multiple frames to indicate temporal stability, the blurred regions, esp. visible on the right side of Fig. 6a ), indicate erroneous motions in the output. For this network the difficulties of temporally changing data and varying output sizes additionally lead to a suboptimal approximation of the target points, that is also visible in terms of an increased L S loss in Table 1. While Fig. 6b ) significantly reduces motions, and leads to an improved shape as well as L S loss, its motions are overly constrained. E.g., at the bottom of the shown patch, the generated points should follow the black input points, but in (b) the generated points stay in place. In addition, the lack of permutation invariance leads to an undesirable clustering of generated points in the patch center. Both problems are removed with L EV in Fig. 6c ), which still contains small scale jittering motions, unfortunately. These are removed by L EA in Fig. 6d), which shows the of our full algorithm. The success of our approach for dynamic output sizes is also shown in in the L N column of Table 1, which contains an L 2 error w.r.t. ground truth size of the outputs. Temporally Coherent Features A central goal of our work is to enable the learning of features that remain stable over time. To shed light on how our approach influences the established latent space, we analyze its content for different inputs. The latent space in our case consists of a 256-dimensional vector that contains the features extracted by the first set of layers of our network. Fig. 8 contains a qualitative example for 100 randomly selected patch sequences from our test data set, where we collect input data by following the trajectory of each patch center for 50 time steps to extract coherent data sets. Fig. 8a ) shows the averaged latent space content over time for these sequences. While the model trained with temporal coherence (green curve) is also visually smoother, the difference becomes clearer when considering temporal frequencies. We measure averaged frequencies of the latent space dimensions over time, as shown in Fig. 8b,c). We quantify the differences by calculating the integral of the frequency spectrumf, weighted by the frequency x to emphasize high frequencies, i.e, x x ·f (x)dx. Hence, small values are better. As shown in Fig. 8b ), the version trained without our loss formulations contains significantly more high frequency content. This is also reflected in the weighted integrals, which are 36.56 for the method without temporal loss, and 16.98 for the method with temporal loss. To verify that our temporal model actually establishes a stable temporal latent space instead of ignoring temporal information altogether, we evaluate the temporal frequencies for the same 100 inputs as above, but with a randomized order over time. In this case, our model correctly identifies the incoherent inputs, and yields similarly high frequencies as the regular model with 28.44 and 35.24, respectively. More details in Appendix C. In addition, we evaluated the changes of generated outputs over time w.r.t. ground truth motion. For this we mapped the generated point cloudsỸ t = {y t 1, y t 2, ..., y t n} for 100 frames to evenly and dense sampled ground-truth points on the original mesh Y t = {y t 1, y t 2, ..., y t n} (the moving man shown in Fig. 1). This gives us a dense correlation between the data and the generated point clouds. For the mapping we used an assignment based on nearest neighbors. γ:ỹ → y. Using γ we divideỸ t into n subsetsŶ i = {ỹ j |γ(ỹ j) = y i } which correlate with the corresponding ground-truth points. For each subset we can now compute the mean position 1 |Ŷi| ŷ∈Ŷiŷ and the sample density |Ŷ i | measured by the number of points assigned to a ground-truth sample position. The temporal change of these values are of particular interest. The change of the mean positions should correspond to the ground-truth changes, while the change of the density should be as small as possible. We have evaluated the error of the first and second derivative of positions, as well as the first and second derivative of density (see Fig. 9 and Table 3). As can be seen from the plots, our method leads to clear improvements for all measured quantities. The individual spikes that are visible for both versions in the position errors (c,d) most likely correspond to sudden changes of the input motions for which our networks undershoots by producing a smooth version of the motion. Table 3: Measurements averaged over 100 frames for a version of our network without temporal loss ("w/o") and with our full temporal loss formulation ("with"). The left table shows the for the error evaluation of the velocity and the acceleration, whereas in the right table one can see the variance of the density derivatives. Our patch-based approach currently relies on a decomposition of the input volumes into patches over time, as outlined in Appendix A. As all of the following involve temporal data, full sequences are provided in the accompanying video. We apply our method to several complex 3D models to illustrate its performance. Fig. 7 shows the input as well as several frames generated with our method for an animation of a spider. Our method produces an even and temporally stable reconstruction of the object. In comparison, Fig. 7b) shows the output from the previous work architecture. It exhibits uneven point distributions and outliers, e.g., above the legs of the spider, in addition to uneven motions. A second example for a moving human figure is shown in Fig. 1. In both examples, our network covers the target shape much more evenly despite using much fewer points, as shown in Table 2. Thanks to the flexible output size of our network, it can adapt to sparsely covered regions by generating correspondingly fewer outputs. The previous work architecture, with its fixed output size, needs to concentrate the fixed number of output points within the target shape, leading to an unnecessarily large point count. In order to demonstrate the flexibility of our method, we also apply it to a volumetric moving point cloud obtained from a liquid simulation. Thanks to the patch-based evaluation of our network, it is agnostic to the overall size of the input volume. In this way, it can be used to generate coherent sets with millions of points. These examples also highlight our method's capabilities for generalization. While the 3D model was only trained on data from physics simulations, as outlined above, it learns stable features that can be flexibly applied to volumetric as well as to surface-based data. The metrics in Table 1 show that for both 2D and 3D cases, our method leads to significantly improved quality, visible in lower loss values for spatial as well as temporal terms. Another interesting field of application for our algorithm are physical simulations. Complex simulations such as fluids, often employ particle-based representations. On the one hand, the volume data is much larger than surface-based data, which additionally motivates our dynamic output. On the other hand, time stability plays a very important role for physical phenomena. Our method produces detailed outputs for liquids, as can be seen in our supplemental video. Convergence graphs for the different versions are shown in Fig. 12 of the supplemental material. These graphs show that our method not only successfully leads to very low errors in terms of temporal coherence, but also improves spatial accuracy. The final values of L S for the 2D case are below 0.05 for our algorithm, compared to almost 0.08 for previous work. For 3D, our approach yields 0.04 on average, in contrast to ca. 0.1 for previous work. We have proposed a first method to infer temporally coherent features for point clouds. This is made possible by a combination of a novel loss function for temporal coherence in combination with enabling flexible truncation of the . In addition we have shown that it is crucial to prevent static patterns as easy-to-reach local minima for the network, which we avoid with the proposed a mingling loss term. Our super-resolution above demonstrate that our approach takes an important first step towards flexible deep learning methods for dynamic point clouds. Looking ahead, our method could also be flexibly combined with other network architectures or could be adopted for other applications. Specifically, a combination with PSGN could be used to generate point clouds from image sequences instead of single images. Other conceivable applications could employ methods like with our approach for generating animated meshes. Due to the growing popularity and ubiquity of scanning devices it will, e.g., be interesting to investigate classification tasks of 3D scans over time as future work. Apart from that, physical phenomena such as elastic bodies and fluids can likewise be represented in a Lagrangian manner, and pose interesting challenges and complex spatio-temporal changes. Data Generation We employ a physical simulation to generate our input and output pairs for training. This has the advantage that it leads a large variety of complex motions, and gives full control of the generation process. More specifically, we employ the IISPH algorithm, a form of Lagrangian fluid simulator that efficiently generates incompressible liquid volumes. These simulations also have the advantage that they inherently control the density of the point sampling thanks to their volume conserving properties. In order to generate input pairs for training, we randomly sample regions near the surface and extract points with a given radius around a central point. This represents the high-resolution target. To compute the low-resolution input, we downsample the points with a Poisson-disk sampling to compute a point set with the desired larger spacing. In order to prevent aliasing from features below the coarse resolution, we perform a pass of surface fairing and smoothing before downsampling. Due to the large number of patches that can be extracted from these simulations, we did not find it necessary to additionally augment the generated data sets. Examples of the low-and high-resolution pairs are shown in the supplemental material. Below we will demonstrate that models trained with this data can be flexibly applied to moving surface data as well as new liquid configurations. The surface data is generated from animated triangle meshes that were resampled with bicubic interpolation in order to match a chosen average per-point area. This pattern was generated once and then propagated over time with the animation. When applying our method to new liquid simulations, we do not perform any downsampling, but rather use all points of a low-resolution simulation directly, as a volumetric re-sampling over time is typically error prone, and gives incoherent low resolution inputs. Given a moving point cloud, we decompose it into temporally coherent patches in the following manner: We start by sampling points via a Poisson-disk sampling in a narrow band around the surface, e.g., based on a signed distance function computed from the input cloud. These points will persist as patch centers over time, unless they move too close to others, or too far away from the surface, which triggers their deletion. In addition, we perform several iterations for every new frame to sample new patches for points in the input cloud that are outside all existing patches. Note that this resampling of patches over time happens instantaneously in our implementation. While a temporal fading could easily be added, we have opted for employing transitions without fading, in order to show as much of the patch content as possible. Network Architecture and Training Our architecture heavily relies on a form of hierarchical point-based convolutions. I.e., the network extracts features for a subset of the points and their nearest neighbors. For the point convolution, we first select a given number of group centers that are evenly distributed points from a given input cloud. For each group center, we then search for a certain number of points within a chosen radius (a fraction of the [-1,1] range). This motivates our choice for a coordinate far outside the regular range for the padded points from Sec. 3.2. They are too far away from all groups by construction, so they are filtered out without any additional overhead. In this way, both feature extraction and grouping operations work flexibly with the varying input sizes. Each group is then processed by a PointNet-like sub-structure , yielding one feature vector per group. The is a set of feature vectors and the associated group position, which can be interpreted as a new point cloud to repeatedly apply a point convolution. In this way, the network extracts increasingly abstract and global features. The last set of features is then interpolated back to the original points of the input. Afterwards a sub-pixel convolution layer is used to scale up the point cloud extended with features and finally the final position vectors are generated with the help of two additional shared, fully connected layers. While we keep the core network architecture unmodified to allow for comparisons with previous work, an important distinction of our approach is the input and output masking, as described in Sec. 3.2. Our point data was generated with a mean point spacing, i.e., Poisson disk radius, of 0.5 units. For the 2D tests, an upscaling factor of r = 9 was used. For this purpose, patches with a diameter of 5 were extracted from the low-resolution data and patches with a diameter of 15 from the high-resolution data. We used the thresholds k max = 100 and n max = 900. For the loss, we used γ = 10, µ = 10, and ν = 0.001. The network was trained with 5 epochs for a data set with 185k pairs, and a batch size of 16, the learning rate was 0.001 with a decay of 0.003. For the 3D below, the scaling factor r was set to 8. The diameter of the patches was 6 for the low-resolution data and 12 for the high-resolution data, with k max = 1280 and n max = 10240. The loss parameters were γ = µ = 5, with ν = 0.001. Learning rate and decay were the same for training, but instead we used 10 epochs with 54k patches in 3D, and a batch size of 4. The input feature vector is processed in the first part of our network, which consists of four point convolutions. We use (n g, r g, [l 1, ..., l d]) to represent a level with n g groups of radius r g and [l 1, ..., l d] the d fully connected layers with the width l i (i = 1, ..., d). The parameters we use are (k max, 0.25, ), (k max /2, 0.5, ), (k max /4, 0.6, ) and (k max /8, 0.7, ). We then use interpolation layers to distribute the features of each convolution level among the input points. In this step, we reduce the output of each convolution layer with one shared, fully-connected layer per level, to a size of 64 and then distribute the features to all points of the input point cloud depending on their position. This extends the points of our original point cloud with 256 features. Fig. 11 shows a visual overview of the data flow in our network. Afterwards, we process the data in r separate branches consisting of two shared, fully interconnected layers with 256 and 128 nodes. The output is then processed with two shared fully connected layers of 64 and 3 nodes. Finally, we add our ing data to the input positions that have been repeated r times. This provides an additional skip connection which leads to slightly more stable . All convolution layers and fully interconnected layers use a tanh activation function. For the input feature vector, we make use of additional data fields in conjunction with the point positions. Our network also accepts additional features such as velocity, density and pressure of the SPH simulations used for data generation. For inputs from other sources, those values could be easily computed with suitable SPH interpolation kernels. In practice, we use position, velocity and pressure fields. Whereas the first two are important (as mentioned in Sec. 3.1), the pressure fields turned out to have negligible influence. In this section we give details for the frequency evaluation of Sec. 4. In order to measure the stability of the latent space against temporal changes, we evaluated the latent space of our network with and without temporal loss, once for 100 ordered patch sequences and once for 100 un-ordered ones. The central latent space of our network consists of the features generated by the point-convolution layers in the first part of the network and is 256 dimensional (see Fig. 11). To obtain information about its general behavior, we average the latent space components over all 100 patch sequences, subtract the mean, and normalize the ing vector w.r.t. maximum value for each data set. The is a time sequence of scalar values representing the mean deviations of the latent space. The Fourier transform of these vectorsf, are shown in Fig. 8, and were used to compute the weighted frequency content Figure 10: Examples from our synthetic data generation process. In (a,b) each a high resolution reference frame is shown in purple, and in green the down-sampled low resolution frames generated from it. The training data is generated by sampled patches from these volumes. Figure 11: An overview of our network architecture. The first row shows the hierarchical point convolutions, while the bottom rows illustrate the processing of extracted features until the final output point coordinates are generated. Figure 12: Convergence plots for the training runs of our different 2D and 3D versions. The combined loss only illustrates convergence behavior for each method separately, as weights and terms differ across the four variants. LM for previous work is not minimized, and only given for reference.

represent the short summary for finding the corresponding scientific article (long)

We propose a generative neural network approach for temporally coherent point clouds.

We propose nearest neighbor overlap, a procedure which quantifies similarity between embedders in a task-agnostic manner, and use it to compare 21 sentence embedders.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We study the problem of generating adversarial examples in a black-box setting in which only loss-oracle access to a model is available. We introduce a framework that conceptually unifies much of the existing work on black-box attacks, and demonstrate that the current state-of-the-art methods are optimal in a natural sense. Despite this optimality, we show how to improve black-box attacks by bringing a new element into the problem: gradient priors. We give a bandit optimization-based algorithm that allows us to seamlessly integrate any such priors, and we explicitly identify and incorporate two examples. The ing methods use two to four times fewer queries and fail two to five times less than the current state-of-the-art. The code for reproducing our work is available at https://git.io/fAjOJ. Recent research has shown that neural networks exhibit significant vulnerability to adversarial examples, or slightly perturbed inputs designed to fool the network prediction. This vulnerability is present in a wide range of settings, from situations in which inputs are fed directly to classifiers BID23 BID3 to highly variable real-world environments BID12. Researchers have developed a host of methods to construct such attacks BID7 BID17 BID2 BID15, most of which correspond to first order (i.e., gradient based) methods. These attacks turn out to be highly effective: in many cases, only a few gradient steps suffice to construct an adversarial perturbation. A significant shortcoming of many of these attacks, however, is that they fundamentally rely on the white-box threat model. That is, they crucially require direct access to the gradient of the classification loss of the attacked network. In many real-world situations, expecting this kind of complete access is not realistic. In such settings, an attacker can only issue classification queries to the targeted network, which corresponds to a more restrictive black box threat model. Recent work BID4 BID1 ) provides a number of attacks for this threat model. BID4 show how to use a basic primitive of zeroth order optimization, the finite difference method, to estimate the gradient from classification queries and then use it (in addition to a number of optimizations) to mount a gradient based attack. The method indeed successfully constructs adversarial perturbations. It comes, however, at the cost of introducing a significant overhead in terms of the number of queries needed. For instance, attacking an ImageNet BID21 classifier requires hundreds of thousands of queries. Subsequent work improves this dependence significantly, but still falls short of fully mitigating this issue (see Section 4.1 for a more detailed analysis). We revisit zeroth-order optimization in the context of adversarial example generation, both from an empirical and theoretical perspective. We propose a new approach for generating black-box adversarial examples, using bandit optimization in order to exploit prior information about the gradient, which we show is necessary to break through the optimality of current methods. We Table 1: Summary of effectiveness of 2 and ∞ ImageNet attacks on Inception v3 using NES, bandits with time prior (Bandits T), and bandits with time and data-dependent priors (Bandits T D). Note that in the first column, the average number of queries is calculated only over successful attacks, and we enforce a query limit of 10,000 queries. For purposes of direct comparison, the last column calculates the average number of queries used for only the images that NES (previous SOTA) was successful on. Our most powerful attack uses 2-4 times fewer queries, and fails 2-5 times less often. Avg Adversarial examples are natural inputs to a machine learning system that have been carefully perturbed in order to induce misbehaviour of the system, under a constraint on the magnitude of the pertubation (under some metric). For image classifiers, this misbehaviour can be either classification as a specific class other than the original one (the targeted attack) or misclassification (the untargeted attack). For simplicity and to make the presentation of the overarching framework focused, in this paper we restrict our attention to the untargeted case. Both our algorithms and the whole framework can be, however, easily adapted to the targeted setting. Also, we consider the most standard threat model in which adversarial perturbations must have p -norm, for some fixed p, less than some p. Suppose that we have some classifier C(x) with a corresponding classification loss function L(x, y), where x is some input and y its corresponding label. In order to generate a misclassified input from some input-label pair (x, y), we want to find an adversarial example x which maximizes L(x, y) but still remains p -close to the original input. We can thus formulate our adversarial attack problem as the following constrained optimization task: DISPLAYFORM0 First order methods tend to be very successful at solving the problem despite its non-convexity BID7 BID2 BID15. A first order method used as the backbone of some of the most powerful white-box adversarial attacks for p bounded adversaries is projected gradient descent (PGD). This iterative method, given some input x and its correct label y, computes a perturbed input x k by applying k steps of the following update (with x 0 = x) DISPLAYFORM1 Here, Π S is the projection onto the set S, B p (x, ε) is the p ball of radius ε around x, η is the step size, and ∂U is the boundary of a set U. Also, as is standard in continuous optimization, we make s l be the projection of the gradient ∇ x L(x l−1, y) at x l−1 onto the unit p ball. This way we ensure that s l corresponds to the unit p -norm vector that has the largest inner product with ∇ x L(x l−1, y).(Note that, in the case of the 2 -norm, s l is simply the normalized gradient but in the case of, e.g., the ∞ -norm, s l corresponds to the sign vector, sgn (∇ x L(x l−1, y)) of the gradient.) So, intuitively, the PGD update perturbs the input in the direction that (locally) increases the loss the most. Observe that due to the projection in, x k is always a valid perturbation of x, as desired. The projected gradient descent (PGD) method described above is designed to be used in the context of so-called white-box attacks. That is, in the setting where the adversary has full access to the gradient ∇ x L(x, y) of the loss function of the attacked model. In many practical scenarios, however, this kind of access is not available-in the corresponding, more realistic black-box setting, the adversary has only access to an oracle that returns for a given input (x, y), only the value of the loss L(x, y).One might expect that PGD is thus not useful in such black-box setting. It turns out, however, that this intuition is incorrect. Specifically, one can still estimate the gradient using only such value queries.(In fact, this kind of estimator is the backbone of so-called zeroth-order optimization frameworks BID22 .) The most canonical primitive in this context is the finite difference method. This method estimates the directional derivative DISPLAYFORM0 Here, the step size δ > 0 governs the quality of the gradient estimate. Smaller δ gives more accurate estimates but also decreases reliability, due to precision and noise issues. Consequently, in practice, δ is a tunable parameter. Now, we can just use finite differences to construct an estimate of the gradient. To this end, one can find the d components of the gradient by estimating the inner products of the gradient with all the standard basis vectors e 1,..., e d: DISPLAYFORM1 We can then easily implement the PGD attack (c.f.) using this estimator: DISPLAYFORM2 Indeed, BID4 were the first to use finite differences methods in this basic form to power PGD-based adversarial attack in the black-box setting. This basic attack was shown to be successful but, since its query complexity is proportional to the dimension, its ing query complexity was prohibitively large. For example, the Inception v3 BID24 classifier on the ImageNet dataset has dimensionality d=268,203 and thus this method would require 268,204 queries. (It is worth noting, however, that BID4 developed additional methods to, at least partially, reduce this query complexity.) In the light of the above discussion, one can wonder if the algorithm can be made more queryefficient. A natural idea here would be to avoid fully estimating the gradient and rely instead only on its imperfect estimators. This gives rise to the following question: How accurate of an gradient estimate is necessary to execute a successful PGD attack?We examine this question first in the simplest possible setting: one in which we only take a single PGD step (i.e., the case of k = 1). Previous work BID7 indicates that such an 0% 5% 10% 15% 20% 25% 30% 35% 40% k percent of ImageNet coordinates 0.2 0.4 0.6 0.8 adversariality rate random-k top-k FIG1: The fraction of correctly estimated coordinates of sgn(∇ x L(x, y)) required to successfully execute the single-step PGD (also known as FGSM) attack, with = 0.05. In the experiment, for each k, the top k percent -chosen either by magnitude (top-k) or randomly (random-k) -of the signs of the coordinates are set correctly, and the rest are set to +1 or −1 at random. The adversariality rate is the portion of 1,000 random ImageNet images misclassified after one FGSM step. For example, estimating only 20% of coordinates correctly leads to misclassification for > 60% of images.attack can already be quite powerful. So, we study how the effectiveness of this attack varies with gradient estimator accuracy. Our experiments, shown in FIG1, suggest that it is feasible to generate adversarial examples without estimating correctly even most of the coordinates of the gradient. For example, in the context of ∞ attacks, setting a randomly selected 20% of the coordinates in the gradient to match the true gradient (and making the remaining coordinates have random sign) is sufficient to fool the classifier on more than 60% images with single-step PGD. Our experiments thus demonstrate that an adversary is likely to be able to cause a misclassification by performing the iterated PGD attack, even when driven by a gradient estimate that is largely imperfect. The above discussion makes it clear that successful attacks do not require a perfect gradient estimation, provided this estimate is suitably constructed. It is still unclear, however, how to efficiently find this kind of imperfect but helpful estimator. Continuous optimization methodology suggests that the key characteristic needed from our estimator is for it to have a sufficiently large inner product with the actual gradient. We thus capture this challenge as the following gradient estimation problem: Definition 1 (Gradient estimation problem). For an input/label pair (x, y) and a loss function L, let g * = ∇ x L(x, y) be the gradient of L at (x, y). Then the goal of the gradient estimation problem is to find a unit vector g maximizing the inner product DISPLAYFORM0 from a limited number of (possibly adaptive) function value queries L(x, y). (The expectation here is taken over the randomness of the estimation algorithm.)One useful perspective on the above gradient estimation problem stems from casting the recovery of g * in as an underdetermined vector estimation task. That is, one can view each execution of the finite difference method (see) as computing an inner product query in which we obtain the value of the inner product of g * and some chosen direction vector A i. Now, if we execute k such queries, and k < d (which is the regime we are interested in), the information acquired in this process can be expressed as the following (underdetermined) linear regression problem Ag * = y, where the rows of the matrix A correspond to the queries A 1,..., A k and the entries of the vector y gives us the corresponding inner product values. Relation to compressive sensing. The view of the gradient estimation problem we developed bears striking similarity to the compressive sensing setting BID5. Thus one might wonder if the toolkit of that area could be applied here. Compressive sensing crucially requires, however, certain sparsity structure in the estimated signal (here, in the gradient g *) and, to our knowledge, the loss gradients do not exhibit such a structure. (We discuss this further in Appendix B.)The least squares method. In light of this, we turn our attention to another classical signal-processing method: norm-minimizing 2 least squares estimation. This method approaches the estimation problem posed in by casting it as an undetermined linear regression problem of the form Ag * = b, where we can choose the matrix A (the rows of A correspond to inner product queries with g *). Then, it obtains the solution g to the regression problem by solving: min DISPLAYFORM1 A reasonable choice for A (via BID11 and related ) is the distancepreserving random Gaussian projection matrix, i.e. A ij normally distributed. The ing algorithm turns out to yield solutions that are approximately those given by Natural Evolution Strategies (NES), which ) previously applied to black-box attacks. In particular, in Appendix A, we prove the following theorem. Theorem 1 (NES and Least Squares equivalence). Letx N ES be the Gaussian k-query NES estimator of a d-dimensional gradient g and letx LSQ be the minimal-norm k-query least-squares estimator of g. For any p > 0, with probability at least 1 − p we have that DISPLAYFORM2 Note that when we work in the underdetermined setting, i.e., when k d (which is the setting we are interested in), the right hand side bound becomes vanishingly small. Thus, the equivalence indeed holds. In fact, using the precise statement (given and proved in Appendix A), we can show that Theorem 1 provides us with a non-vacuous equivalence bound. Further, it turns out that one can exploit this equivalence to prove that the algorithm proposed in Ilyas et al. FORMULA1 is not only natural but optimal, as the least-squares estimate is an information-theoretically optimal gradient estimate in the regime where k = d, and an error-minimizing estimator in the regime where k << d. Theorem 2 (Least-squares optimality (Proof in Appendix A)). For a linear regression problem y = Ag with known A and y, unknown g, and isotropic Gaussian errors, the least-squares estimator is finite-sample efficient, i.e. the minimum-variance unbiased (MVU) estimator of the latent vector g. Theorem 3 (Least-squares optimality (Proof in Meir FORMULA1). In the underdetermined setting, i.e. when k << d, the minimum-norm least squares estimate (x LSQ in Theorem 1) is the minimumvariance (and thus minimum-error, since bias is fixed) estimator with no empirical loss. The optimality of least squares strongly suggests that we have reached the limit of query-efficiency of black-box adversarial attacks. But is this really the case? Surprisingly, we show that an improvement is still possible. The key observation is that the optimality we established of least-squares (and by Theorem 1, the NES approach in) holds only for the most basic setting of the gradient estimation problem, a setting where we assume that the target gradient is a truly arbitrary and completely unknown vector. However, in the context we care about this assumption does not hold -there is actually plenty of prior knowledge about the gradient available. Firstly, the input with respect to which we compute the gradient is not arbitrary and exhibits locally predictable structure which is consequently reflected in the gradient. Secondly, when performing iterative gradient attacks (e.g. PGD), the gradients used in successive iterations are likely to be heavily correlated. The above observations motivate our focus on prior information as an integral element of the gradient estimation problem. Specifically, we enhance Definition 1 by making its objective DISPLAYFORM0, where I is prior information available to us. This change in perspective gives rise to two important questions: does there exist prior information that can be useful to us?, and does there exist an algorithmic way to exploit this information? We show that the answer to both of these questions is affirmative. Consider a gradient ∇ x L(x, y) of the loss function corresponding to some input (x, y). Does there exist some kind of prior that can be extracted from the dataset {x i}, in general, and the input (x, y) in particular, that can be used as a predictor of the gradient? We demonstrate that it is indeed the case, and give two example classes of such priors. Time-dependent priors. The first class of priors we consider are time-dependent priors, a standard example of which is what we refer to as the "multi-step prior." We find that along the trajectory taken by estimated gradients, successive gradients are in fact heavily correlated. We show this empirically by taking steps along the optimization path generated by running the NES estimator at each point, and plotting the normalized inner product (cosine similarity) between successive gradients, given by Figure 2 demonstrates that there indeed is a non-trivial correlation between successive gradientstypically, the gradients of successive steps (using step size from) have a cosine similarity of about 0.9. Successive gradients continue to correlate at higher step sizes: Appendix B shows that the trend continues even at step size 4.0 (a typical value for the total perturbation bound ε). This indicates that there indeed is a potential gain from incorporating this correlation into our iterative optimization. To utilize this gain, we intend to use the gradients at time t − 1 as a prior for the gradient at time t, where both the prior and the gradient estimate itself evolve over iterations. DISPLAYFORM0 Data-dependent priors. We find that the time-dependent prior discussed above is not the only type of prior one can exploit here. Namely, we can also use the structure of the inputs themselves to reduce query complexity (in fact, the existence of such data-dependent priors is what makes machine learning successful in the first place).In the case of image classification, a simple and heavily exploited example of such a prior stems from the fact that images tend to exhibit a spatially local similarity (i.e. pixels that are close together tend to be similar). We find that this similarity also extends to the gradients: specifically, whenever two coordinates (i, j) and DISPLAYFORM1 To corroborate and quantify this phenomenon, we compare ∇ x L(x, y) with an average-pooled, or "tiled", version (with "tile length" k) of the same signal. An example of such an average-blurred gradient can be seen in Appendix B. More concretely, we apply to the gradient the mean pooling operation with kernel size (k, k, 1) and stride (k, k, 1), then upscale the spatial dimensions by k. We then measure the cosine similarity between the average-blurred gradient and the gradient itself. Our , shown in Figure 3, demonstrate that the gradients of images are locally similar enough to allow for average-blurred gradients to maintain relatively high cosine similarity with the actual gradients, even when the tiles are large. Our suggest that we can reduce the dimensionality of our problem by a factor of k 2 (for reasonably large k) and still estimate a vector pointing close to the same direction as the original gradient. This factor, as we show later, leads to significantly improved black-box adversarial attack performance. Given the availability of these informative gradient priors, we now need a framework that enables us to easily incorporate these priors into our construction of black-box adversarial attacks. Our proposed method builds on the framework of bandit optimization, a fundamental tool in online convex optimization BID9. In the bandit optimization framework, an agent plays a game that consists of a sequence of rounds. In round t, the agent must choose a valid action, and then by playing the action incurs a loss given by a loss function t (·) that is unknown to the agent. After playing the action, he/she only learns the loss that the chosen action incurs; the loss function is specific to the round t and may change arbitrarily between rounds. The goal of the agent is to minimize the average loss incurred over all rounds, and the success of the agent is usually quantified by comparing the total loss incurred to that of the best expert in hindsight (the best single-action policy). By the nature of this formulation, the rounds of this game can not be treated as independent -to perform well, the agent needs to keep track of some latent record that aggregates information learned over a sequence of rounds. This latent record usually takes a form of a vector v t that is constrained to a specified (convex) set K. As we will see, this aspect of the bandit optimization framework will provide us with a convenient way to incorporate prior information into our gradient prediction. An overview of gradient estimation with bandits. We can cast the gradient estimation problem as an bandit optimization problem in a fairly direct manner. Specifically, we let the action at each round t be a gradient estimate g t (based on our latent vector v t), and the loss t correspond to the (negative) inner product between this prediction and the actual gradient. Note that we will never have a direct access to this loss function t but we are able to evaluate its value on a particular prediction vector g t via the finite differences method (which is all that the bandits optimization framework requires us to be able to do).Just as this choice of the loss function t allows us to quantify performance on the gradient estimation problem, the latent vector v t will allow us to algorithmically incorporate prior information into our predictions. Looking at the two example priors we consider, the time-dependent prior will be reflected by carrying over the latent vector between the gradient estimations at different points. Data-dependent priors will be captured by enforcing that our latent vector has a particular structure. For the specific prior we quantify in the preceding section (data-dependent prior for images), we will simply reduce the dimensionality of the latent vector via average-pooling ("tiling"), removing the need for extra queries to discern components of the gradient that are spatially close. We now describe our bandit framework for adversarial example generation in more detail. Note that the algorithm is general and can be used to construct black-box adversarial examples where the perturbation is constrained to any convex set (p -norm constraints being a special case). We discuss the algorithm in its general form, and then provide versions explicitly applied to the 2 and ∞ cases. As previously mentioned, the latent vector v t ∈ K serves as a prior on the gradient for the corresponding round t -in fact, we make our prediction g t be exactly v t projected onto the appropriate space, and thus we set K to be an extension of the space of valid adversarial perturbations (e.g. R n for 2 examples, [−1, 1] n for ∞ examples). Our loss function t is defined as DISPLAYFORM0 for a given gradient estimate g, where we access this inner product via finite differences. Here, L(x, y) is the classification loss on an image x with true class y. The crucial element of our algorithm will thus be the method of updating the latent vector v t. We will adapt here the canonical "reduction from bandit information" BID9. Specifically, our update procedure is parametrized by an estimator ∆ t of the gradient ∇ v t (v), and a first-order update step DISPLAYFORM1, which maps the latent vector v t and the estimated gradient of t with respect to v t (which we denote ∆ t) to a new latent vector v t+1. The ing general algorithm is presented as Algorithm 1.In our setting, we make the estimator ∆ of the gradient −∇ v ∇L(x, y), v of the loss be the standard spherical gradient estimator (see BID9). We take a two-query estimate of the expectation, and employ antithetic sampling which in the estimate being computed as DISPLAYFORM2 Algorithm 1 Gradient Estimation with Bandit Optimization DISPLAYFORM3 for each round t = 1,..., T do // Our loss in round t is t (g t) = − ∇ x L(x, y init), g t g t ← v t−1 6: DISPLAYFORM0 where u is a Gaussian vector sampled from N (0, {q 1, q 2} ← {v + δu, v − δu} // Antithetic samples 4: DISPLAYFORM1 // Note that due to cancellations we can actually evaluate ∆ with only two queries to L return ∆ A crucial point here is that the above gradient estimator ∆ t parameterizing the bandit reduction has no direct relation to the "gradient estimation problem" as defined in Section 2.4. It is simply a general mechanism by which we can update the latent vector v t in bandit optimization. It is the actions g t (equal to v t) which provide proposed solutions to the gradient estimation problem from Section 2.4.The choice of the update rule A tends to be natural once the convex set K is known. For K = R n, we can simply use gradient ascent: DISPLAYFORM0 and the exponentiated gradients (EG) update when the constraint is an ∞ bound (i.e. K = [−1, 1] n ): DISPLAYFORM1 Finally, in order to translate our gradient estimation algorithm into an efficient method for constructing black-box adversarial examples, we interleave our iterative gradient estimation algorithm with an iterative update of the image itself, using the boundary projection of g t in place of the gradient (c.f. FORMULA1). This in a general, efficient, prior-exploiting algorithm for constructing black-box adversarial examples. The ing algorithm in the 2 -constrained case is shown in Algorithm 3. We evaluate our bandit approach described in Section 3 and the natural evolutionary strategies (NES) approach of on their effectiveness in generating untargeted adversarial examples. We consider both the 2 and ∞ threat models on the ImageNet BID21 dataset, in terms of success rate and query complexity. We further investigate loss and gradient estimate quality over the optimization trajectory in each method. To show the method extends to other datasets, DISPLAYFORM0 x 0 ← x init // Adversarial image to be constructed 5:while C(x) = y init do 6: DISPLAYFORM1 7: ∆ t ← GRAD-EST(x t−1, y init, v t−1) // Estimated Gradient of t DISPLAYFORM2 v t ← v t−1 + η · ∆ t 10: DISPLAYFORM0 we also compare to NES in the CIFAR-∞ threat model; in all threat models, we show on Inception-v3, Resnet-50, and VGG16 classifiers. In evaluating our approach, we test both the bandit approach with time prior (Bandits T), and our bandit approach with the given examples of both the data and time priors (Bandits T D). We use 10,000 and 1,000 randomly selected images (scaled to) to evaluate all approaches on ImageNet and CIFAR-10 respectively. For NES, Bandits T, and Bandits T D we found hyperparameters (given in Appendix C, along with the experimental parameters) via grid search. For ImageNet, we record the effectiveness of the different approaches in both threat models in Table 1 (2 and ∞ perturbation constraints), where we show the attack success rate and the mean number of queries (of the successful attacks) needed to generate an adversarial example for the Inception-v3 classifier ( for other classifiers in Appendix F). For all attacks, we limit the attacker to at most 10,000 oracle queries. As shown in Table 1, our bandits framework with both data-dependent and time prior (Bandits T D), is six and three times less failure-prone than the previous state of the art (NES) in the ∞ and 2 settings, respectively. Despite the higher success rate, our method actually uses around half as many queries as NES. In particular, when restricted to the inputs on which NES is successful in generating adversarial examples, our attacks are 2.5 and 5 times as query-efficient for the ∞ and 2 settings, respectively. In Appendix G, we also compare against the AutoZOOM method of BID25, where we show that our Bandits T D method at a higher 100% success rate is over 6 times as query-efficient. Finally, we also have similar for CIFAR-10 under the ∞ threat model, which can be found in Appendix E.We also further quantify the performance of our methods in terms of black-box attacks, and gradient estimation. Specifically, we first measure average queries per success after reaching a certain success rate (Figure 4a), which indicates the dependence of the query count on the desired success rate. The data shows that for any fixed success rate, our methods are more query-efficient than NES, and (due to the exponential trend) suggest that the difference may be amplified for higher success rates. We then plot the loss of the classifier over time (averaged over all images), and performance on the gradient estimation problem for both ∞ and 2 cases (which, crucially, corresponds directly to the expectation we maximize in. We show these three plots for ∞ in Figure 4, and show the for 2 (which are extremely similar) in Appendix D, along with CDFs showing the success of each method as a function of the query limit. We find that on every metric in both threat models, our methods strictly dominate NES in terms of performance. All known techniques for generating adversarial examples in the black-box setting so far rely on either iterative optimization schemes (our focus) or so-called substitute networks and transferability. In the first line of work, algorithms use queries to gradually perturb a given input to maximize a corresponding loss, causing misclassification. BID19 presented the first such iterative attack on a special class of binary classifiers. Later, BID26 Figure 4: (left) Average number of queries per successful image as a function of the number of total successful images; at any desired success rate, our methods use significantly less queries per successful image than NES, and the trend suggests that this gap increases with the desired success rate. (center) The loss over time, averaged over all images; (right) The correlation of the latent vector with the true gradient g, which is precisely the gradient estimation objective we define.real-world system with black-box attacks. Specifically, they fool PDF document malware classifier by using a genetic algorithms-based attack. Soon after, described the first black-box attack on deep neural networks; the algorithm uses a greedy search algorithm that selectively changes individual pixel values. BID4 were the first to design black-box attack based on finite-differences and gradient based optimization. The method uses coordinate descent to attack black-box neural networks, and introduces various optimizations to decrease sample complexity. Building on the work of BID4, designed a black-box attack strategy that also uses finite differences but via natural evolution strategies (NES) to estimate the gradients. They then used their algorithm as a primitive in attacks on more restricted threat models. In a concurrent line of work, BID20 introduce a method for attacking models with so-called substitute networks. Here, the attacker trains a model -called a substitute network -to mimic the target network's decisions (obtained with black-box queries), then uses (white-box) adversarial examples for the substitute network to attack the original model. Adversarial examples generated with these methods BID20; BID14 tend to transfer to a target MNIST or CIFAR classifier. We note, however, that for attacking single inputs, the overall query efficiency of this type of methods tends to be worse than that of the gradient estimation based ones. Substitute models are also thus far unable to make targeted black-box adversarial examples. We develop a new, unifying perspective on black-box adversarial attacks. This perspective casts the construction of such attacks as a gradient estimation problem. We prove that a standard least-squares estimator both captures the existing state-of-the-art approaches to black-box adversarial attacks, and actually is, in a certain natural sense, an optimal solution to the problem. We then break the barrier posed by this optimality by considering a previously unexplored aspect of the problem: the fact that there exists plenty of extra prior information about the gradient that one can exploit to mount a successful adversarial attack. We identify two examples of such priors: a "time-dependent" prior that corresponds to similarity of the gradients evaluated at similar inputs, and a "data-dependent" prior derived from the latent structure present in the input space. Finally, we develop a bandit optimization approach to black-box adversarial attacks that allows for a seamless integration of such priors. The ing framework significantly outperforms state-of-the-art by a factor of two to six in terms of success rate and query efficiency. Our thus open a new avenue towards finding priors for construction of even more efficient black-box adversarial attacks. We thank Ludwig Schmidt for suggesting the connection between LSQ and NES. AM supported in part by NSF grants CCF-1553428 and CNS-1815221. LE supported in part by a Siebel Foundation Scholarship and IBM Watson AI grant. AI supported by an Analog Devices Fellowship. Theorem 1 (NES and Least Squares equivalence). Letx N ES be the Gaussian k-query NES estimator of a d-dimensional gradient g and letx LSQ be the minimal-norm k-query least-squares estimator of g. For any p > 0, with probability at least 1 − p we have that DISPLAYFORM0 and in particular, DISPLAYFORM1 with probability at least 1 − p, where DISPLAYFORM2 Proof. Let us first recall our estimation setup. We have k query vectors δ i ∈ R d drawn from an i.i.d Gaussian distribution whose expected squared norm is one, i.e. δ i ∼ N (0, DISPLAYFORM3 Let the vector y ∈ R k denote the inner products of δ i s with the gradient, i.e. DISPLAYFORM4 We define the matrix A to be a k × d matrix with the δ i s being its rows. That is, we have Ag = y. Now, recall that the closed forms of the two estimators we are interested in are given bŷ DISPLAYFORM5 which implies that DISPLAYFORM6 We can bound the difference between these two inner products as DISPLAYFORM7 Now, to bound the first term in, observe that DISPLAYFORM8 and thus DISPLAYFORM9 (Note that the first term in the above sum has been canceled out.) This gives us that DISPLAYFORM10 as long as AA T − I ≤ 1 2 (which, as we will see, is indeed the case with high probability). Our goal thus becomes bounding AA T − I = λ max (AA T − I), where λ max (·) denotes the largest (in absolute value) eigenvalue. Observe that AA T and −I commute and are simultaneously diagonalizable. As a , for any 1 ≤ i ≤ k, we have that the i-th largest eigenvalue λ i (AA T − I) of AA T − I can be written as DISPLAYFORM11 So, we need to bound DISPLAYFORM12 To this end, recall that E[AA T] = I (since the rows of A are sampled from the distribution N (0, 1 d I)), and thus, by the covariance estimation theorem of BID6 (see Corollary 7.2) (and union bounding over the two relevant events), we have that DISPLAYFORM13 and thus DISPLAYFORM14 with probability at least 1 − k k+1 p. To bound the second term in, we note that all the vectors δ i are chosen independently of the vector g and each other. So, if we consider the set {ĝ,δ 1, . . .,δ k} of k + 1 corresponding normalized directions, we have (see, e.g., BID8) that the probability that any two of them have the (absolute value of) their inner product be larger than some ε = 2 log(2(k+1)/p) d is at most DISPLAYFORM15.On the other hand, we note that each δ i is a random vector sampled from the distribution N (0, DISPLAYFORM16, so we have that (see, e.g., Lemma 1 in BID13), for any 1 ≤ i ≤ k and any ε > 0, DISPLAYFORM17.Theorem 2 (Least-Squares Optimality). For a fixed projection matrix A and under the following observation model of isotropic Gaussian noise: y = Ag + ε where ε ∼ N (0, εId), the least-squares estimator as in Theorem 1,x LSQ = A T (AA T) −1 y is a finite-sample efficient (minimum-variance unbiased) estimator of the parameter g. Proving the theorem requires an application of the Cramer-Rao Lower Bound theorem:Theorem 3 (Cramer-Rao Lower Bound). Given a parameter θ, an observation distribution p(x; θ), and an unbiased estimatorθ that uses only samples from p(x; θ), then (subject to Fisher regularity conditions trivially satisfied by Gaussian distributions), DISPLAYFORM0 Now, note that the Cramer-Rao bound implies that if the variance of the estimatorθ is the inverse of the Fisher matrix,θ must be the minimum-variance unbiased estimator. Recall the following form of the Fisher matrix: DISPLAYFORM1 Now, suppose we had the following equality, which we can then simplify using the preceding equation: DISPLAYFORM2 Multiplying the preceding by [I(θ)] −1 on both the left and right sides yields: DISPLAYFORM3 which tells us that is a sufficient condition for finite-sample efficiency (minimal variance). We show that this condition is satisfied in our case, where we have y ∼ Ag + ε,θ =x LSQ, and θ = g. We begin by computing the Fisher matrix directly, starting from the distribution of the samples y: DISPLAYFORM4 ∂ log p(y; g) DISPLAYFORM5 Using FORMULA1, DISPLAYFORM6 Finally, note that we can write: DISPLAYFORM7 which concludes the proof, as we have shown thatx LSQ satisfies the condition, which in turn implies finite-sample efficiency. Claim 1. Applying the precise bound that we can derive from Theorem 1 on an ImageNet-sized dataset (d = 300000) and using k = 100 queries (what we use in our ∞ threat model and ten times that used for our 2 threat model), DISPLAYFORM8 For 10 queries, DISPLAYFORM9 B OMITTED FIGURES Compressed sensing approaches can, in some cases, solve the optimization problem presented in Section 2.4. However, these approaches require sparsity to improve over the least squares method. Here we show the lack of sparsity in gradients through a classifier on a set of canonical bases for images. In FIG4, we plot the fraction of 2 weight accounted for by the largest k components in randomly chosen image gradients when using two canonical bases: standard and wavelet (db4). While lack of sparsity in these bases does not strictly preclude the existence of a basis on which gradients are sparse, it suggests the lack of a fundamental structural sparsity in gradients through a convolutional neural network. We show in FIG6 that the correlation between successive gradients on the NES trajectory are signficantly correlated, even at much higher step sizes (up to 2 norm of 4.0, which is a typical value for ε, the total adversarial perturbation bound and thus an absolute bound on step size). This serves as further motivation for the time-dependent prior. The average number of queries used per successful image for each method when reaching a specified success rate: we compare NES, Bandits T (our method with time prior only), and Bandits T D (our method with both data and time priors) and find that our methods strictly dominate NES-that is, for any desired sucess rate, our methods take strictly less queries per successful image than NES. Here, we give for the CIFAR-10 dataset, comparing our best method (Bandits T D) and NES. We train Inception-v3, ResNet-50, and VGG16 classifiers by fine-tuning the standard PyTorch ImageNet classifiers. As such, all images are upsampled to 224 × 224 (299 × 299) for. Just as for ImageNet, we use a maximum ∞ perturbation of 0.05, where images are scaled to. Table 5: Summary of effectiveness of ∞ CIFAR10 attacks on Inception v3, ResNet-50, and VGG16 (I, R, V) using NES and bandits with time and data-dependent priors (Bandits T D). Note that in the first column, the average number of queries is calculated only over successful attacks, and we enforce a query limit of 10,000 queries. For purposes of direct comparison, the last column calculates the average number of queries used for only the images that NES (previous SOTA) was successful on. Our most powerful attack uses 2-4 times fewer queries, and fails 2-22 times less often. Table 1 ), VGG16, and ResNet50 classifiers. Note that we do not fine-tune the hyperparameters to the new classifiers, but simply use the hyperparameters found for Inception-v3. Nevertheless, our best method consistently outperforms NES on black-box attacks. Table 6: Summary of effectiveness of ∞ and 2 ImageNet attacks on Inception v3, ResNet-50, and VGG16 (I, R, V) using NES and bandits with time and data-dependent priors (Bandits T D). Note that in the first column, the average number of queries is calculated only over successful attacks, and we enforce a query limit of 10,000 queries. For purposes of direct comparison, the last column calculates the average number of queries used for only the images that NES (previous SOTA) was successful on. Our most powerful attack uses 2-4 times fewer queries, and fails 2-5 times less often. To compare with the method of BID25, we consider the same classifier and dataset (Inceptionv3 and Imagenet) under the same 2 threat model. Note that BID25 use mean rather than maximum 2 perturbation to evaluate their attacks (since the method is based on a Lagrangian relaxation). To ensure a fair comparison we compare against the average number of queries to reach the adversarial examples bounded within a pertubation budget of 2 · 10 −4, which is explicitly reported byTu et al..For the bandits approach, we used Bandits T, (the bandits method with the time prior) and Bandits T D (the bandits method with both time and data prior) and run the methods until 100% success is reached. We use the same hyperparameters from the untargeted ImageNet experiments (given in Appendix C). Our findings, given in Table 7 show that our best method achieves an 100% success rate, and an over 6-fold reduction in queries. Note that the method of BID25 achieves 100% success rate in general, but only constrains the mean 2 perturbation, and thus actually achieves a strictly less than 100% success rate with this perturbation threshold. Table 7: Comparison against coordinate-based query efficient finite differences attacks from BID25, using the ImageNet dataset, with a maximum 2 constraint of 0.0002 per-pixel normalized (which is equal to a max-2 threshold reported by BID25). For our methods (Bandits T and Bandits T D) we use the same hyperparameters as in our comparison to NES, which are given in Appendix C. Avg. Queries Success Rate AutoZOOM-BiLin BID25 15,064 <100% AutoZOOM-AE BID25 14,914 <100% Bandits T (Ours) 4455 100% Bandits T D (Ours) 2297 100%

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We present a unifying view on black-box adversarial attacks as a gradient estimation problem, and then present a framework (based on bandits optimization) to integrate priors into gradient estimation, leading to significantly increased performance.

represent the short summary for finding the corresponding scientific article (long)!

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the article for finding a single-sentence TLDR

Question: Collecting high-quality, large scale datasets typically requires significant resources. The aim of the present work is to improve the label efficiency of large neural networks operating on audio data through multitask learning with self-supervised tasks on unlabeled data. To this end, we trained an end-to-end audio feature extractor based on WaveNet that feeds into simple, yet versatile task-specific neural networks. We describe three self-supervised learning tasks that can operate on any large, unlabeled audio corpus. We demonstrate that, in a scenario with limited labeled training data, one can significantly improve the performance of a supervised classification task by simultaneously training it with these additional self-supervised tasks. We show that one can improve performance on a diverse sound events classification task by nearly 6\% when jointly trained with up to three distinct self-supervised tasks. This improvement scales with the number of additional auxiliary tasks as well as the amount of unsupervised data. We also show that incorporating data augmentation into our multitask setting leads to even further gains in performance. Although audio tag classification does not require the fine temporal resolution found in raw audio 91 waveforms, our chosen auxiliary tasks (or any arbitrary auditory task for which we may desire our 92 model to be sufficient) require higher temporal resolutions. To satisfy this, we chose to build our 93 model following the WaveNet architecture. are processed using small, task-specific neural networks built atop a task-agnostic trunk. The trunk architecture principally follows the structure of WaveNet, with several blocks of stacked, dilated, and causal convolutions between every convolution layer. Outputs from the trunk are fed into task-specific heads (details in Section 3.1).As shown Figure 1, our WaveNet trunk is composed of N blocks, where each block consists of S dilated causal convolution layers, with dilation factors increasing from 1 to 2 S − 1, residual connections and saturating nonlinearities. We label the blocks using b = 1, · · ·, N. We use indices ∈ [1 + (b − 1)S, bS] to label layers in block b. Each layer,, of the WaveNet trunk consists of a "residual atom" which involves two computations, labeled as "Filter" and "Gate" in the figure. Each residual atom computation produces a hidden state vector h and a layer output x defined via DISPLAYFORM0 where denotes element-wise products, represents the regular convolution operation, denotes dilated convolutions with a dilation factor of 2 mod bS if is a layer in block b + 1, σ denotes the sigmoid function and W gate and W f ilter are the weights for the gate and filter, respectively. The first (= 0) layer -represented as the initial stage marked "1 × 1 Conv" in Figure 1 -applies causal convolutions to the raw audio waveforms X = (X 1, X 2, · · ·, X T), sampled at 16 kHz, to produce an output DISPLAYFORM1 Given the structure of the trunk laid out above, any given block b has an effective receptive field of 1 + b(2 S − 1). Thus the total effective receptive field of our trunk is τ = 1 + N (2 S − 1). Following an extensive hyperpameter search over various configurations, we settled on [N = 3] blocks comprised of [S = 6] layers each for our experiments. Thus our trunk has a total receptive field of τ = 190, which corresponds to about 12 milliseconds of audio sampled at 16kHz. As indicated above, each task-specific head is a simple neural network whose input data is first 102 constrained to pass through a trunk that it shares with other tasks. Each head is free to process this 103 input to its advantage, independent of the other heads. Each task also specifies its own objective function, as well as a task-specific optimizer, with cus-105 tomized learning rates and annealing schedules, if necessary. We arbitrarily designate supervised 106 tasks as the primary tasks and refer to any self-supervised tasks as auxiliary tasks. In the experiments 107 reported below, we used "audio tagging" as the primary supervised classification task and "next-step 108 prediction", "noise reduction" and "upsampling" as auxiliary tasks training on various amounts of unlabeled data. The parameters used for each of the task specific heads can be found in TAB1 accompanying supplement to this paper. Figure 2: The head architectures were designed to be simple, using only as few layers as necessary to solve the task. Simpler head architectures force the shared trunk to learn a representation suitable for multiple audio tasks. The next-step prediction task can be succinctly formalized as follows: given a sequence 113 {x t−τ +1, · · ·, x t} of frames of an audio waveform, predict the next value x t+1 in the sequence. This prescription allows one to cheaply obtain arbitrarily large training datasets from an essentially 115 unlimited pool of unlabeled audio data. Our next-step prediction head is a 2-layer stack of 1×1 convolutional layers with ReLU nonlinearities 117 in all but the last layer. The first layer contains 128 units, while the second contains a single output unit. The head takes in τ frames of data from the trunk, where τ is the trunk's effective receptive field, and 119 produces an output which represents the model's prediction for the next frame of audio in the sequence. The next-step head treats this as a regression problem, using the mean squared error of the difference 121 between predicted values and actual values as a loss function, i.e. given inputs {x t−τ +1, · · ·, x t}, the head produces an output y t from which we compute a loss L MSE (t) = (y t − x t+1) 2 and then 123 aggregate over the frames to get the total loss. We would like to note that the original WaveNet implementation treated next-step prediction as a 125 classification problem, instead predicting the bin-index of the audio following a µ-law transform. We 126 found that treating the task as a regression problem worked better in multitask situations but make no claims on the universality of this choice. In defining the noise reduction task, we adopt the common approach of treating noise as an additive 130 random process on top of the true signal: if {x t} denotes the clean raw audio waveform, we obtain 131 the noisy version viax t:= x t + ξ t where ξ t an arbitrary noise process. For the denoising task, the 132 model is trained to predict the clean sample, x t, given a window x t− well-adapted to solving either task. Thus, our noise reduction head has a structure similar to the 136 next-step head. It is trained to minimize a smoothed L1 loss between the clean and noisy versions of 137 the waveform inputs, i.e. for each frame t, the head produces an outputŷ t, and we compute the loss DISPLAYFORM0 and then aggregate over frames to obtain the total loss. We used the smooth L1 loss because it 139 provided a more stable convergence for the denoising task than mean squared error. In the same spirit as the denoising task, one can easily create an unsupervised upsampling task Again, given the formal similarity of the upsampling task to the next-step prediction and noise-151 reduction tasks, we used an upsampling head with a structure virtually identical to those described 152 above. As with the denoising task, we used a smooth L1 loss function (see eqn. FORMULA2 having manually-verified labels and the remaining 5763 having non-verified labels, meaning they 177 were automatically categorized using user-provided metadata. The test set is composed of 1600 audio 178 clips with manually-verified labels which are used for the final scoring. The Librispeech dataset 1 (comprised of read English speech sampled at 16 kHz) was used as a proxy for a large unlabeled dataset. The models described below were trained using clips from either the 182 "train-clean-100" or "train-other-500 versions". Models trained with 5, 50 and 100 hours of unlabeled 183 data were sourced from "train-clean-100", while the model trained with 500 hours was sourced entirely from "train-other-500". Due to memory constraints, we limited the duration of each utterance 185 to 2 seconds which we obtained by cropping from a random position in the original clip. This dataset 186 was only used to train the auxiliary tasks. We trained the model using raw audio waveform inputs taken from the FSDKaggle2018 and Lib-189 rispeech datasets. All code for the experiments described here was written in the PyTorch framework. All audio samples were first cropped to two seconds in duration and downsampled to 16 kHz. To normalize for the variation in onset times for different utterances, the 2 seconds were randomly both the main task and the auxiliary tasks, heuristically favoring performance on the main task. We jointly trained the model on all tasks simultaneously by performing a forward pass for each task, important parameters of the model can be found in TAB4 of the accompanying supplement to this 217 paper. As discussed above, we used audio tagging as the main task to investigate whether supervised First, we trained a purely supervised model on 2 seconds of non-silence audio extracted using random In this experiment, we added each of the self-supervised tasks to the baseline model discussed above, simultaneously training them using 100 hours of unlabeled data sampled from the Librispeech dataset 236 along with the main supervised task. We notice that, addition of any self-supervised task showed an average improvement of 4.6% to the MAP@3 score compared to the main task's baseline performance. Adding a pair of tasks gave an average improvement of 4.55% over baseline, showing no improvement over adding a single task. Training with three additional tasks yielded the best with an improvement of 5.33% over the main task. Looking at MAP@3 scores throughout training showed that convergence in every multitask setting was stable, with gradual improvements for increasing number of tasks. The best performance values on the test sets for a sequence of task additions can be found in TAB1.The set of experiments described above demonstrate that, for a fixed amount of unlabeled data (100 hours), simultaneously training a supervised task with various self-supervised tasks yields a significant improvement in the main task's performance. To further test how performance changes with increasing amounts of data, we re-trained our model while varying the amount of unlabeled data used to train the auxiliary tasks. We noticed that even without any additional unlabeled data, the MAP@3 score with three additional tasks was significantly better than the score obtained on a single task. This demonstrates that addition of self-supervised tasks improves the performance of main task. Increasing the size of the unlabeled data for the auxiliary tasks increases the size of the multitask benefit (Figure 3).The MAP@3 Scores at different levels of unlabaled data showed progressive improvement to 0.656, 0.669, with 5 and 50 hours respectively. We observed a peak MAP@3 score of 0.694 with 500 hours of unlabeled data, which is an improvement of 8.94% over the main task's baseline performance. Next, we explore several approaches to data augmentation and compare them with multitask learning. Previous work has demonstrated the effectiveness of data augmentation through simple techniques, such as noise injection, and pitch shifting. We compared our proposed method with traditional data augmentation strategies by retraining our model only for the main task after applying the aforementioned augmentations to the FSDKaggle2018 training data. The MAP@3 values for the data augmentation experiments on the test sets can be found in TAB2. We observed a peak MAP@3 score of 0.703 with pitch shifting augmentation which is similar in scale to that of our best multitask performance gains. In an attempt to observe how both the techniques work together, we combined data augmentation with multitask learning and obtain an MAP@3 score of 0.726 which was the best score among all the experiments we conducted. Figure 3: Improved MAP@3 scores with increasing amounts of unlabeled data. Shown are the MAP@3 scores on test set when the main task is trained with 3 auxiliary tasks with 0, 5, 50, 100, and 500 hours of unlabeled data respectively. The amount of labelled data is held constant for the whole experiment. We see a smooth increase in performance with increasing amounts of unlabeled data. where one has a limited quantity of labeled data. We have also shown that the performance of the 244 supervised task improves by increasing either the number of auxiliary self-supervised tasks or the 245 quantity of unlabeled data or both. We attain a peak performance boost of 8.94% over the baseline 246 with the inclusion of 3 self-supervised tasks when trained with additional 500 hours of unlabeled data. Finally, our multitask learning scheme further benefits when the training data for the data-constrained 248 task is augmented using standard techniques. Since our suggest that the performance gain with 249 our approach is additive when used with data augmentation, it may be interesting to use multitask 250 learning with other augmentation approaches to observe if they complement each other in different 251 settings. We have strived to systematically present our within a coherent multitask learning framework.

Represent the short summary for finding the corresponding scientific article (long)

Improving label efficiency through multi-task learning on auditory data

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.