Question: Represent the scientific article to find a short TLDR

Question: We introduce three generic point cloud processing blocks that improve both accuracy and memory consumption of multiple state-of-the-art networks, thus allowing to design deeper and more accurate networks. The novel processing blocks that facilitate efficient information flow are a convolution-type operation block for point sets that blends neighborhood information in a memory-efficient manner; a multi-resolution point cloud processing block; and a crosslink block that efficiently shares information across low- and high-resolution processing branches. Combining these blocks, we design significantly wider and deeper architectures. We extensively evaluate the proposed architectures on multiple point segmentation benchmarks (ShapeNetPart, ScanNet, PartNet) and report systematic improvements in terms of both accuracy and memory consumption by using our generic modules in conjunction with multiple recent architectures (PointNet++, DGCNN, SpiderCNN, PointCNN). We report a 9.7% increase in IoU on the PartNet dataset, which is the most complex, while decreasing memory footprint by 57%. Figure 1: (Left) Memory footprint and inference speed of network variations: our multiresolution (mRes), and crosslink (X) blocks decrease the memory footprint, while our convolutiontype block (conv) decreases both memory consumption (-67%) and inference time (-41%) compared to the PointNet++ (PN++) baseline. (Right) Improvements in accuracy for three segmentation benchmarks of increasing complexity. On the -most complex-PartNet dataset our deep network outperforms the shallow PointNet++ baseline by 3.4%(spread of +0.6), yielding a 9.7% (spread of +3.4) relative increase. Geometry processing has recently started profiting from applying deep learning to graphics and 3D shape analysis (b; b; with networks that guarantee desirable properties of point cloud processing, such as permutation-invariance and quantization-free representation ; . Despite these advances, several differences still impede the breakthroughs made in computer vision. The different nature of 3D data dictates re-inventing for geometry processing the functionality of basic image processing blocks, such as multi-resolution processing or convolution operations. When operating with unstructured point clouds, one has to resort to elementary local pooling operations that group information within a neighborhood based on Euclidean distance. Exemplary methods such as the PointNet/PointNet++ architectures (a; make design choices that potentially compromise performance. In particular, the computation and memory demands of point network blocks can affect both training speed and, more crucially, inference time. One of the main bottlenecks for point networks is their memory-intensive nature: as detailed in Sec. 3.1, the PointNet++ architecture and its variants replicate point neighborhood information, letting every node carry in its feature vector information about all of its neighborhood. This in significant memory overhead, and limits the number of layers, features and feature compositions one can compute. In this work, we enhance point processing networks by introducing a set of modules that improve memory footprint and accuracy, without compromising on inference speed. We call the architectures Lean Point Networks, to highlight their lightweight memory budget. We build on the decreased memory budget to go deeper with point networks. As has been witnessed repeatedly in the image domain ;), we show that going deep also increases the prediction accuracy of point networks. We start in Sec. 3.2 by replacing the grouping operation used in point cloud processing networks with a low-memory alternative that is the point cloud processing counterpart of efficient image processing implementations of convolution. The ing'point convolution block' is 67% more memory-efficient and 41% faster than its PointNet++ counterpart, while exhibiting favorable training properties due to more effective mixing of information across neighborhoods. We then turn in Sec. 3.3 to improving the information flow across layers and scales within point networks through three techniques: a multi-resolution variant for multi-scale network which still delivers the multi-scale context but at a reduced memory and computational cost, residual links, and a new cross-link block that broadcasts multi-scale information across the network branches. By combining these advances we are able to successfully train deeper point networks that allow us to leverage upon larger, recently introduced datasets. In Sec. 4 we thoroughly validate our contributions on the ShapeNet-Part, ScanNet and PartNet segmentation benchmarks, reporting systematic improvements over the PointNet++ baseline. As shown in Fig. 1, when combined these contributions deliver multifold reductions in memory consumption while improving performance, allowing us in a second stage to train increasingly wide and deep networks. On PartNet, the most complex dataset, our deep architecture achieves a 9.7% relative increase in IoU while decreasing memory footprint by 57% and inference time by 47%. Having thoroughly ablated our design choices on the PartNet++ baseline, in Sec. 4.3 we turn to confirming the generic nature of our blocks. We extend the scope of our experiments to three additional networks, (i) DGCNN (b), (ii) SpiderCNN and (iii) PointCNN (b) and report systematic improvements in memory efficiency and performance. Learning in Point Clouds. Learning-based approaches have recently attracted significant attention in the context of Geometric Data Analysis, with several methods proposed specifically to handle point cloud data, including PointNet (a) and several extensions such as PointNet++ (b) and Dynamic Graph CNNs (b) for shape segmentation and classification, PCPNet for normal and curvature estimation, P2P-Net and PU-Net (b) for cross-domain point cloud transformation. Although many alternatives to PointNet have been proposed a;; ) to achieve higher performance, the simplicity and effectiveness of PointNet and its extension PointNet++ make it popular for many other tasks (a). Taking PointNet++ as our starting point, our work facilitates the transfer of network design techniques developed in computer vision to point cloud processing. In particular, significant accuracy improvements have been obtained with respect to the original AlexNet network by engineering the scale of the filtering operations , the structure of the computational blocks , and the network's width and depth ). A catalyst for experimenting with a larger space of network architecture, however, is the reduction of memory consumption -this motivated us to design lean alternatives to point processing networks. introduce a new operator to improve point cloud network efficiency, but only focus on increasing the convergence speed by tuning the receptive field. has investigated how residual/dense connections and dilated convolution could help mitigate vanishing gradient observed for deep graph convolution networks but without solving memory limitations. By contrast our work explicitly tackles the memory problem with the objective of training deeper/wider networks and shows that there are clear improvements over strong baselines. Memory-Efficient Networks. The memory complexity of the standard back-propagation implementation grows linearly with network's depth as backprop requires retaining in memory all of the intermediate activations computed during the forward pass, since they are required for the gradient computation in the backward pass. Several methods bypass this problem by trading off speed with memory. Checkpointing techniques use anchor points to free up intermediate computation , and re-compute them in the backward pass. This is 1.5x slower during training, since one performs effectively two forward passes rather than just one. More importantly, applying this technique is easy for chain-structured graphs, e.g., recursive networks but is not as easy for general Directed Acyclic Graphs, such as U-Nets, or multi-scale networks like PointNet++. One needs to manually identify the graph components, making it cumbersome to experiment with diverse variations of architectures. Reversible Residual Networks (RevNets) limit the computational block to come in a particular, invertible form of residual network. This is also 1.5x slower during training, but alleviates the need for anchor points altogether. Unfortunately, it is unclear what is the point cloud counterpart of invertible blocks. We propose generic blocks to reduce the memory footprint inspired from multi-resolution processing and efficient implementations of the convolution operation in computer vision. As we show in Sec. 4.3, our blocks can be used as drop-in replacements in generic point processing architectures (PointNet++, DGCNN, SpiderNet, PointCNN) without any additional network design effort. This Section introduces a set of modular blocks, shown in Fig. 2, that can be applied to most state-ofthe-art point networks. We start with a brief introduction of the PointNet++ network, which serves as an example point network baseline. We then introduce our modules and explain how their design decreases memory footprint and improves information flow. PointNet++ (b) builds on top of PointNet. First, each point p i looks up its k-nearest neighbors and stacks them to get a point set, say P i N k. Then, PointNet is applied to each such point set P i N k and the ant feature vector is assigned back to the corresponding point p i. While demonstrated to be extremely effective, PointNet++ has two main shortcomings: first, because of explictly carrying around k-nearest neighbor information for each point, the network layers are memory intensive; and second, being reliant on PointNet, it also delays transmission of global information until the last stage. As shown in Fig. 2(a), the existing PointNet++ grouping operation exposes the neighborhood of any point i by concatenating all of its K neighboring D-dimensional vectors v [i,k] to form a tensor: Every vector of this matrix is processed separately by a Multi-Layer-Perceptron that implements a function MLP: R D → R D, while at a later point a max-pooling operation over the K neighbors of every point delivers a slim, N × D matrix. When training a network every layer constructs and retains such a matrix in memory, so that it can be used in the backward pass to update the MLP parameters, and send gradients to earlier layers. The counterpart for a standard 2D image convolution amounts to forming a K 2 tensor in memory when performing K × K filtering and then implementing a convolution as matrix multiplication. This amounts to the im2col operation used for example in the caffe library to implement convo- The standard PN++ layer in (a) amounts to the composition of a neighborhood-based lookup and a PointNet element. In (b) we propose to combine parallel PointNet++ blocks in a multiresolution architecture, and in (c) allow information to flow across branches of different resolutions through a cross-link element. In (d) we propose to turn the lookup-SLP-pooling cascade into a low-memory counterpart by removing the kNN elements from memory once computed; we also introduce residual links, improving the gradient flow. In (e) we stack the block in (d) to grow in depth and build our deep architecture. Each of these tweaks to the original architecture allows for systematic gains in memory and computational efficiency. The green box indicates that the block can be grown in depth by stacking those green units. lutions with General Matrix-Matrix Multiplication (GEMM) . In point clouds the nearest neighbor information provides us with the counterpart to the K ×K neighborhood. Based on this observation we propose to use the same strategy as the one used in memory-efficient implementations of image convolutions for deep learning. In particular we free the memory as soon as the forward pass computes its output, rather than maintaining the matrix in memory. In the backward pass we reconstruct the matrix on the fly from the outputs of the previous layer. We perform the required gradient computations and then return the GPU memory resources (we refer to Algorithm 1 and Algorithm 2 in Appendix B for a detailed description). Using the on-the-fly re-computation of the tensor T has a positive impact on both the forward and backward pass. Instead of applying the SLP on the neighbourhood feature matrix, we can first apply the SLP on the flat feature matrix and then reconstruct the neighbourhood just before the max-pooling layer. The same can be used for the backward pass. In our unoptimized code, our convolution-type architecture shortens the time spent for the forward pass and the backward pass by 41% and 68% respectively on average. For a network with L layers, the memory consumption of the baseline PointNet++ layer grows as L×(N ×D×K), while in our case memory consumption grows as L×(N ×D)+(N ×D×K), where the term, L × (N × D) accounts for the memory required to store the layer activations, while the second term N ×D ×K is the per-layer memory consumption of a single neighborhood convolution layer. As L grows larger, this in a K-fold drop, shown on Fig. 3. This reduction opens up the possibility of learning much deeper networks, since memory demands now grow substantially more slowly in depth. With minor, dataset-dependent, fluctuations, the memory footprint of our convolution type architecture is on average 67% lower than the PointNet++ baseline, while doubling the number of layers comes with a memory overhead of 2.7%. Figure 3: Evolution of memory consumption as the number of layers increases for PointNet++ and convPN (convolution block counterpart) on ShapeNet-Part. Doubling the number of layers for convPN only in an increase in memory by +2.3% and +16.8% for mid-and high-resolution respectively, which favorably compares to the +72% and +125% increases for PointNet++. We now turn to methods that allow for a more efficient propagation of information through the network. As has been repeatedly shown in computer vision, this can drastically impact the behavior of the network during training. Our experiments indicate that this is also the case for point processing. Due to lack of space, the illustrations clarifying the operation of these blocks are only available in the Appendix. a-Multi-Resolution vs Multi-Scale Processing. Shape features can benefit from both local, finegrained information and global, semantic-level context; their fusion can easily boost the discriminative power of the ing features. As presented in Sec. 3.1, PointNet++ mostly uses a neighborhood of fixed radius, leaving only to the later blocks of the network the use of bigger radii and point set sampling. Hence, at early stages of the network, the points only has access to very local information. We observe that this allows only very slow exchange of information among low-, mid-and coarsescale information. Coarse-scale information is conveyed not necessarily by all points that are contained within a larger radius, but by obtaining potentially sparse measurements from a larger area. This underlies also the common log-polar sampling mechanism in computer and biological vision (; ;) where a constant number of measurements is obtained in concentric disks of geometrically increasing radii. We therefore propose to extract neighborhoods of fixed size in downsampled versions of the original point cloud. In the coordinates of the original point cloud this amounts to increasing the effective grouping area, but it now comes with a much smaller memory budget. We observe a 58% decrease in memory footprint on average on the three tested datasets. Please refer to Fig. 4 in Appendix A.1.2 for an illustration of the difference between both types of processing. b-Residual Links. We use the standard Residual Network architecture, which helps to train deep networks reliably. Residual networks change the network's connectivity to improve gradient flow during training: identity connections provide early network layers with access to undistorted versions of the loss gradient, effectively mitigating the vanishing gradient problem. As our in Sec. 4 show, this allows us to train deeper networks. c-Cross Links. We further introduce Cross-Resolution Links in order to better propagate information in the network during training. We draw inspiration from the Multi-Grid Networks and the Multiresolution Tree Networks and allow layers that reside in different resolution branches to communicate with each other, thereby exchanging low-, mid-, and high-resolution information throughout the network processing, rather than fusing multi-resolution information at the end of each block. Cross-links broadcast information across resolutions as shown in Fig. 5 in Appendix A.1: unlike , an MLP transforms the output of one branch to the right output dimensionality so that it can be combined with the output of another branch. Each resolution branch can focus on its own representation and the MLPs will be in charge of making the translation between them. Taking in particular the case of a high-resolution branch communicating its outputs to a mid-resolution branch, we have N × D H feature vectors at the output of a lookup-SLP-pooling block cascade, which need to be communicated to the N/2 × D M vectors of the mid-resolution branch. We first downsample the points, going from N to N/2 points, and then use an MLP that transforms the vectors to the target dimensionality. Conversely, when going from low-to higher dimensions we first transform the points to the right dimensionality and then upsample them. We have experimented with both concatenating and summing multi-resolution features and have observed that summation behaves systematically better in terms of both training speed and test performance. Dataset and evaluation measures. We evaluate our modules on the point cloud segmentation task on three different datasets. The datasets consist of either 3D CAD models or real-world scans. We quantify the complexity of each dataset based on (i) the number of training samples, (ii) the homogeneity of the samples and (iii) the granularity of the segmentation task. Note that a network trained on a bigger and diverse dataset would be less prone to overfitting -as such we can draw more informative from more complex datasets. We order the datasets by increasing complexity: ShapeNet-Part , ScanNet and PartNet for fine-grained segmentation. By its size (24,506 samples) and its granularity (251 labeled parts), PartNet is the most complex dataset we have experimented on. In order to stay consistent with reported benchmarks on each dataset, we use two different metrics to report the Intersection over Union (IoU): (i) the mean Intersection over Union (mIoU) and (ii) the part Intersection over Union (pIoU). Please refer to Appendix C.2 for further explanation about both metrics. We report the performance of our variations for PointNet++ on the Shapenet-Part, ScanNet and PartNet datasets (Table 1). Our lean and deep architectures can be easily deployed on large and complex datasets. Hence, for PartNet, we choose to train on the full dataset all at once on a segmentation task across the 17 classes instead of having to train a separate network for each category in contrast to the original paper . Our architectures substantially improve the memory efficiency of the PointNet++ baseline while also delivering an increase in performance for more complex datasets (see Fig. 1). Indeed, as the data complexity grows, having efficient information flow has a larger influence on the network performance. On PartNet, the spread between our architectures and the vanilla PointNet++ becomes significantly high: our multiresolution (mRes) network increases relative performance by +5.7% over PointNet++ and this gain reaches +6.5% with cross-links (mResX). Our convolution-type network (convPN) outperforms other architectures when dataset complexity increases (+12.5% on ScanNet and +7.4% on PartNet) by more efficiently mixing information across neighbours. The aforementioned memory savings give the opportunity to design significantly deeper networks. Naively increasing network depth can harm performance; instead, we use residual connections to improve convergence for our deep network. The exact design of this architecture is more thoroughly detailed in Appendix A.1.4 but consists in doubling the number of layers in the encoding part. While keeping the impact on efficiency very small (+6.3% on inference time on average and +3.6% on memory consumption at most compared to the shallow convPN), the performance is improved by a significant margin as shown in Table 1. On PartNet, this margin reaches +2.1% over the shallow convPN and +9.7% over the vanilla PointNet++. We underline the extremely low growth of memory as a function of depth, shown in Fig. 3. In Table 2 we compare against Deep GCN in terms of the overall accuracy and the mean IoU on S3DIS dataset . We attain similar performance to the current state-of-the-art, while relying on our generic memory-efficient network blocks and while based on a weaker baseline compared to Deep GCN (i.e. DGCNN); as we show in the following section, these blocks comes with the advantage of being applicable to many SOTA point processing networks. We have introduced building blocks for point processing networks based on two key ideas, (i) a memory efficient convolution and (ii) a multi-resolution approach. Our blocks make it really efficient to capture, process and diffuse information in a point neighbourhood. Diffusing information across neighborhood is the main behavior that most networks, if not all, share. We validate the generality of the proposed modular blocks in the context of other state-of-the-art point-based learning setups, as shown in Table 3. Each of our macro-blocks can be stacked together, extended into a deeper block by duplicating the green boxes (see Figure 2) or even be modified by changing one of its components by another. We test our framework on three additional networks among the latest state-of-the-art approaches, (i) Dynamic Graph CNN (b), (ii) PointCNN (b) and (iii) SpiderCNN . The efficiency of our modules for the KPConv network is explored in Appendix C.4. These networks involve a diverse set of point convolution approaches and thus allows us to assess the generic nature of our modular blocks. All three of the networks make extensive use of memory which is a bottleneck to depth. We implant our modules directly in the original networks, making, when needed, some approximations from the initial architecture. We report the performance of each network with our lean counterpart on two metrics: (i) memory footprint and (ii) IoU in Table 3. Our lean counterparts consistently improve both the IoU (from +1.0% up to +8.0%) and the memory consumption (from -19% up to -69%). Our modular blocks can thus be applied to a wide range of state-of-the-art networks and improve significantly their memory consumption while having a positive impact on performance. In this section, we report our extensive experiments to assess the importance of each block of our network architectures. Our lean structure allows us to adjust the network architectures by increasing its complexity, either by (i) adding extra connections or by (ii) increasing the depth. We analyze our networks along four axes: (i) the performance measured in IoU (Table 1), (ii) the memory footprint, (iii) the inference time and (iv) the length of the backward pass. Our main experimental findings regarding network efficiency are reported in Table 4 and ablate the impact of our proposed design choices for point processing networks. Table 4: Efficiency of our network architectures measured with a batch size of 8 samples on a Nvidia GTX 2080Ti GPU. All of our lean architectures allow to save a substantial amount of memory on GPU wrt. the PointNet++ baseline from 58% with mRes to a 67% decrease with convPN. This latter convolution-type architecture wins on all counts, decreasing both inference time (-41%) and the length of backward pass (-68%) by a large spread. Starting form this architecture, the marginal cost of going deep is extremely low: doubling the number of layers in the encoding part of the network increases inference time by 6.3% on average and the memory consumption by only 3.6% at most compared to convPN. Please refer to Appendix C.3 for absolute values. Multi-Resolution: Processing different resolutions at the same stage of a network has been shown to perform well in shallow networks. Indeed, mixing information at different resolutions helps to capture complex features early in the network. We adopt that approach to design our mRes architecture. Switching from a PointNet++ architecture to a multi-resolution setting increases the IoU by 1.2% on ShapeNet-Part and 5.7% on PartNet. More crucially, this increase in performance come with more efficiency. Although the inference time is longer (18% longer on average) due to the extra downsampling and upsampling operations, the architecture is much leaner and reduces by 58% the memory footprint. The training time is quicker due to a 62% faster backward pass. Cross-links: Information streams at different resolutions are processed separately and can be seen as complementary. To leverage this synergy, the network is provided with additional links connecting neighborhood resolutions. We experiment on the impact of those cross-resolution links to check their impact on the optimization. At the price of a small impact on memory efficiency (+8% wrt. mRes) and speed (+7% on inference time wrt. mRes), the performance can be improved on PartNet, the most complex dataset, with these extra-links by 0.8%. As described in Sec. 3.2, our leanest architeture is equivalent to constraining each PointNet unit to be composed of a single layer network, and turning its operation into a memory-efficient block by removing intermediate activations from memory. In order to get a network of similar size, multiple such units are stacked to reach the same number of layers as the original network. Our convolution-type network wins on all counts, both on performance and efficiency. Indeed, the IoU is increased by 12.5% on ScanNet and 7.4% on PartNet compared to PointNet++ baseline. Regarding its efficiency, the memory footprint is decreased by 67% on average while decreasing both inference time (-41%) and the length of the backward pass (-68%). These improvements in speed can be seen as the consequence of processing most computations on flatten tensors and thus reducing drastically the complexity compared to PointNet++ baseline. In this work we have introduced new generic building blocks for point processing networks, that exhibit favorable memory, computation, and optimization properties when compared to the current counterparts of state-of-the-art point processing networks. When based on PointNet++, our lean architecture convPN wins on all counts, memory efficiency (-67% wrt. PointNet++) and speed (-41% and -68% on inference time and length of backward pass). Its deep counterpart has a marginal cost in terms of efficiency and achieves the best IoU on PartNet (+9.7% over PointNet++). Those generic and modular blocks exhibit similar performance on all of the additional tested architectures with a significant decrease in memory (up to -69%) and increase in IoU (up to +8.0%). From the promising on PartNet and the extremely low cost of depth in our architectures, we anticipate that adding these components to the armament of the deep geometry processing community will allow researchers to train the next generation of point processing networks by leveraging upon the advent of larger shape datasets (; In this section, we provide more details about how we design our lean architectures to ensure reproducible for all tested architectures, (i) PointNet++ (b), (ii) Dynamic Graph CNN (b), (iii) SpiderCNN and (iv) PointCNN (b). We implement each networks in Pytorch following the original code in Tensorflow and implant our blocks. To keep things simple and concise in this section, we adopt the following notations: • S(n): Sampling layer of n points; • rNN(r): query-ball of radius r; • MaxP: Max Pooling along the neighbourhood axis; •: Multi-resolution combination; • Lin(s): Linear unit of s neurons; • Drop(p): Dropout layer with a probability p to zero a neuron Inside our architectures, every downsampling module is itself based on FPS to decrease the resolution of the input point cloud. To get back to the original resolution, upsampling layers proceed to linear interpolation in the spatial space using the K u closest neighbours. To generate multiple resolutions of the same input point cloud, a downsampling ratio of 2 is used for every additional resolution (see Fig. 5). In all our experiments, we choose to report the performance of the multi-scale PointNet++ (MSG PN++) as it is reported to beat its alternative versions in the original paper on all tasks. We implement our own implementation of PointNet++ in Pytorch and choose the same parameters as in the original code. For segmentation task, the architecture is designed as follow: We omit here skiplinks for sake of clarity: they connect encoding and decoding modules at the same scale level. The mRes architecture consists in changing the way the sampling is done in the network to get a multiresolution approach (Fig. 4). We provide the details only for the encoding part of the network as we keep the decoding part unchanged from PointNet++. Encoding1: Starting from this architecture, we add Xlinks connection between each layer of mLPs to get our mResX architecture. A Xlink connection connects two neighbouring resolutions to merge information at different granularity. On each link, we use a sampling module (either downsampling or upsampling) to match the input to the target resolution. We use two alternatives for feature combination: (a) concatenation, (b) summation. In the later case, we add an additional sLP on each Xlink to map the input feature dimension to the target. To keep this process as lean as possible, we position the SLP at the coarser resolution, i.e. before the upsampling module or after the downsampling module. To simplify the writing, we adopt the additional notations: T ) where we make a sampling of s i points on each resolution i. When only one resolution is available as input, the block S([., s 1, s 2, ..., s n−1] T ) will sequentially downsample the input point cloud by s 1, s 2,.. points to create the desired number of resolutions. • Convolution block C([r 1, r 2, ..., r n] T ) is composed itself of three operations for each resolution i: neighborhood lookup to select the r i NN for each points, an sLP layer of the same size as its input and a max-pooling. • Transition block T ([t 1, t 2, ..., t n] T ) whose main role is to change the channel dimension of the input to one of the convolution block. An sLP of ouput dimension t i will be apply to the resolution i. We also add Xlinks inside each of the C blocks. In this architecture, the features are combined by summation and the links follow the same design as for mResX. Our deep architecture builds on convPN to design a deeper architecture. For our experiments, we double the size of the encoding part by repeating each convolution block twice. For each encoding segment, we position the sampling block after the third convolution block, so that the first half of the convolution blocks are processing a higher resolution point cloud and the other half a coarsen version. Starting from the authors' exact implementation, we swap each edge-conv layer, implemented as an MLP, by a sequence of single resolution convPN blocks. The set of convPN blocks replicates the succession of SLP used in the original implementation (to build their MLPs). To allow the use of residual links, a transition block is placed before each edge-conv layer to match the input dimension of our convPN blocks to their output dimension. A SpiderConv block can be seen as a bilinear operator on the input features and on a non-linear transformation of the input points. This non-linear transformation consists of changing the space where the points live. In the original architecture, an SLP is first applied to the transformed points to compute the points' Taylor expansion. Then, each output vector is multiplied by its corresponding feature. Finally a convolution is applied on the product. Therefore, the neighbourhood features can be built on-the-fly within the block and deleted once the outputs are obtained. We thus modify the backward pass to reconstruct the needed tensors for gradient computation. For PointCNN, we modify the χ-conv operator to avoid having to store the neighbourhood features for the backward pass. To do so, we make several approximations from the original architecture. We replace the first MLP used to lift the points by a sequence of convPN blocks. Thus, instead of learning a feature representation per neighbour, we retain only a global feature vector per representative point. We change as well the first fully connected layer used to learn the χ-transformation matrix. This new layer now reconstructs the neighbourhood features on-the-fly from its inputs and deletes it from memory as soon as its output is computed. During the backward pass, the neighbourhood features tensor is easily rebuilt to get the required gradients. We implement the same trick for the convolution operator applied to the transformed features. We further augment this layer with the task of applying the χ-transformation to the neighbourhood features once grouped. Finally, we place transition blocks between each χ-conv operation to enable residual links. In all our experiments, we process the dataset to have the same number of points N for each sample. To reach a given number of points, input pointclouds are downsampled using the furthest point sampling (FPS) algorithm or randomly upsampled. We keep the exact same parameters as the original networks regarding most of parameters. To regularize the network, we interleave a dropout layer between the last fully connected layers and parameterize it to zero 70% of the input neurons. Finally, we add a weight decay of 5e-4 to the loss for all our experiments. All networks are trained using the Adam optimizer to minimize the cross-entropy loss. The running average coefficients for Adam are set to 0.9 and 0.999 for the gradient and its square, respectively. The 2D counterpart of convolution operates in three steps: (i) neighbourhood exposure, (ii) matrix multiplication and (iii) pooling through a sum operator. Our convPN block follows the same steps but it can be seen as non-standard convolution as each weight matrix is constrained to be identical for all neighbours. To expose the neighbourhood, an intermediate tensor needs to be built to store neighborhood features. This tensor can be then used to gather and refine local information for each patch. This process has been used for image processing as the so-called im2col operation to rearrange discrete image blocks in a two-dimensional tensor. Exposing the neighbourhood simplifies the convolution to a simple matrix multiplication and thus fastens the convolution operation but does have a critical memory footprint if not handled properly. Indeed, neighborhood features as any other activations will be saved into memory to allow gradients to flow downward the graph. We design our memory efficient block to build the neighborhood matrix on-the-fly (see Algorithm 1 and 2) without the need to store neighborhood features. Algorithm 1: Low-memory grouping -Forward pass We evaluate our network on the point cloud segmentation task on three different datasets, ordered by increasing complexity: • ShapeNet-Part : CAD models of 16 different object categories composed of 50 labeled parts. The dataset provides 13, 998 samples for training and 2, 874 samples for evaluation. Point segmentation performance is assessed using the mean point Intersection over Union (mIoU). • ScanNet: Scans of real 3D scenes (scanned and reconstructed indoor scenes) composed of 21 semantic parts. The dataset provides 1, 201 samples for training and 312 samples for evaluation. We follow the same protocol as in Qi et al. (2017a) and report both the accuracy and the part Intersection over Union (pIoU). • PartNet : Large collection of CAD models of 17 object categories composed of 251 labeled parts. The dataset provides 17, 119 samples for training, 2, 492 for validation and 4, 895 for evaluation. The dataset provides a benchmark for three different tasks: fine-grained semantic segmentation, hierarchical semantic segmentation and instance segmentation. We report on the first task to evaluate our networks on a more challenging segmentation task using the same part Intersection over Union (pIoU) as in ScanNet. To report our , we use two versions of the Intersection over Union metric: • mIoU: To get the per sample mean-IoU, the IoU is first computed for each part belonging to the given object category, whether or not the part is in the sample. Then, those values are averaged across the parts. If a part is neither predicted nor in the ground truth, the IoU of the part is set to 1 to avoid this indefinite form. The mIoU obtained for each sample is then averaged to get the final score as, with n samples the number of samples in the dataset, cat(s), n cat(s) parts and P cat(s) the object category where s belongs, the number of parts in this category and the sets of its parts respectively. IoU s (p i) is the IoU of part p i in sample s. • pIoU: The part-IoU is computed differently. The IoU per part is first computed over the whole dataset and then, the values obtained are averaged across the parts as, with n parts the number of parts in the dataset, I s (p i) and U s (p i) the intersection and union for samples s on part p i respectively. To take into account the randomness of point cloud sampling when performing coarsening, we use the average of'N' forward passes to decide on the final segmentation during evaluation. The following section provides more details on the evaluation experiments introduced in the paper. We present the per-class IoU on both ShapeNet-Part (Table 5) and PartNet (Table 6) datasets for each of the PointNet++ based architecture. Due to the high number of points per sample and the level of details of the segmentation, PartNet can be seen as a much more complex than ShapeNet-Part. As additional reference, we provide on Table 8 the performance of our lean blocks applied to three architectures when training one network per-object category on PartNet (on Chairs and Tables that represents 60% of the dataset). On PartNet, the spread between an architecture with an improved information flow and a vanilla one becomes significant. Our PointNet++ based networks perform consistently better than the original architecture on each of the PartNet classes. Increasing the depth of the network allows to achieve a higher accuracy on the most complex classes such as Chairs or Lamps composed of 38 and 40 different part categories respectively. As shown on Fig. 6, our deep architecture is able to better capture the boundaries between parts and thus to predict the right labels very close from part edges. When a sample is itself composed of many parts, having a deep architecture is a significant advantage. For reference, we provide as well the absolute values for the efficiency of each of those networks measured by three different metrics on Table 7: (i) memory footprint, (ii) inference time and (iii) length of backward pass. We provide on Table 9 some additional experiments with KPConv network , a kernel point convolution approach. We report the efficiency with and without our modules, evaluated both in terms of memory consumption and forward/backward speed. Our modules successfully help to reduce the memory by up to 52.5% while having no impact on the speed of the forward or backward pass. Table 7: Efficiency of our network architectures measured with a batch size of 8 samples on a Nvidia GTX 2080Ti GPU. All of our lean architectures allow to save a substantial amount of memory on GPU wrt. the PointNet++ baseline from 58% with mRes to a 67% decrease with convPN. This latter convolution-type architecture wins on all counts, decreasing both inference time (-41%) and the length of backward pass (-68%) by a large spread. Starting form this architecture, the marginal cost of going deep is extremely low: doubling the number of layers in the encoding part of the network increases inference time by 6.3% on average and the memory consumption by only 3.6% at most compared to convPN) Parameters ( Figure 6: Segmentation prediction on the test set for both PointNet++ and our deepConvPN network compared to the ground truth. While PointNet++ struggles to detect accurately the boundaries between different parts, our deep architecture performs a much finer segmentation in those frontier areas.

Represent the TLDR for finding the corresponding scientific article

We introduce three generic point cloud processing blocks that improve both accuracy and memory consumption of multiple state-of-the-art networks, thus allowing to design deeper and more accurate networks.

represent the short summary for finding the corresponding scientific article (long)

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question:
represent the article for finding a single-sentence tldr
------
Given We present a meta-learning approach for adaptive text-to-speech (TTS) with few data. During training, we learn a multi-speaker model using a shared conditional WaveNet core and independent learned embeddings for each speaker. The aim of training is not to produce a neural network with fixed weights, which is then deployed as a TTS system. Instead, the aim is to produce a network that requires few data at deployment time to rapidly adapt to new speakers. We introduce and benchmark three strategies: (i) learning the speaker embedding while keeping the WaveNet core fixed, (ii) fine-tuning the entire architecture with stochastic gradient descent, and (iii) predicting the speaker embedding with a trained neural network encoder. The experiments show that these approaches are successful at adapting the multi-speaker neural network to new speakers, obtaining state-of-the-art in both sample naturalness and voice similarity with merely a few minutes of audio data from new speakers. Training a large model with lots of data and subsequently deploying this model to carry out classification or regression is an important and common methodology in machine learning. It has been particularly successful in speech recognition, machine translation BID1 and image recognition BID2 BID3. In this textto-speech (TTS) work, we are instead interested in few-shot meta-learning. Here the objective of training with many data is not to learn a fixed-parameter classifier, but rather to learn a "prior" neural network. This prior TTS network can be adapted rapidly, using few data, to produce TTS systems for new speakers at deployment time. That is, the intention is not to learn a fixed final model, but rather to learn a model prior that harnesses few data at deployment time to learn new behaviours rapidly. The output of training is not longer a fixed model, but rather a fast learner. Biology provides motivation for this line of research. It may be argued that evolution is a slow adaptation process that has ed in biological machines with the ability to adapt rapidly to new data during their lifetimes. These machines are born with strong priors that facilitate rapid learning. We consider a meta-learning approach where the model has two types of parameters: task-dependent parameters and task-independent parameters. During training, we learn all of these parameters but discard the task-dependent parameters for deployment. The goal is to use few data to learn the task-dependent parameters for new tasks rapidly. Task-dependent parameters play a similar role to latent variables in classical probabilistic graphical models. Intuitively, these variables introduce flexibility, thus making it easier to learn the taskindependent parameters. For example, in classical HMMs, knowing the latent variables in a simple learning problem of estimating the parameters of an exponential-family distribution. In neural networks, this approach also facilitates learning when there is clear data diversity and categorization. We show this for adaptive TTS BID4 BID5. In this setting, speakers correspond to tasks. During training we have many speakers, and it is therefore helpful to have task-dependent parameters to capture speaker-specific voice styles. At the same time, it is useful to have a large model with shared parameters to capture the generic process of mapping text to speech. To this end, we employ the WaveNet model. WaveNet BID6 is an autoregressive generative model for audio waveforms that has yielded state-of-art performance in speech synthesis. This model was later modified for real-time speech generation via probability density distillation into a feed-forward model BID7. A fundamental limitation of WaveNet is the need for hours of training data for each speaker. In this paper we describe a new WaveNet training procedure that facilitates adaptation to new speakers, allowing the synthesis of new voices from no more than 10 minutes of data with high sample quality. We propose several extensions of WaveNet for sample-efficient adaptive TTS. First, we present two non-parametric adaptation methods that involve fine-tuning either the speaker embeddings only or all the model parameters given few data from a new speaker. Second, we present a parametric textindependent approach whereby an auxiliary network is trained to predict new speaker embeddings. The experiments will show that all the proposed approaches, when provided with just a few seconds or minutes of recording, can generate high-fidelity utterances that closely resemble the vocal tract characteristics of a demonstration speaker, particularly when the entire model is fine-tuned end-to-end. When fine-tuning by first estimating the speaker embedding and subsequently fine-tuning the entire model, we achieve state-of-the-art in terms of sample naturalness and voice similarity to target speakers. These are robust across speech datasets recorded under different conditions and, moreover, we demonstrate that the generated samples are capable of confusing the state-of-the-art text-independent speaker verification system BID8.TTS techniques require hours of high-quality recordings, collected in controlled environments, for each new voice style. Given this high cost, reducing the length of the training dataset could be valuable. For example, it is likely to be very beneficial when attempting to restore the voices of patients who suffer from voice-impairing medical conditions. In these cases, long high quality recordings are scarce. WaveNet is an autoregressive model that factorizes the joint probability distribution of a waveform, x = {x 1, . . ., x T}, into a product of conditional distributions using the probabilistic chain rule: DISPLAYFORM0 p(x t |x 1:t−1, h; w), where x t is the t-th timestep sample, and h and w are respectively the conditioning inputs and parameters of the model. To train a multi-speaker WaveNet, the conditioning inputs h consist of the speaker identity s, the linguistic features l, and the logarithmic fundamental frequency f 0 values. l encodes the sequence of phonemes derived from the input text, and f 0 controls the dynamics of the pitch in the generated utterance. Given the speaker identity s for each utterance in the dataset, the Figure 2: Training (slow, lots of data), adaptation (fast, few data) and inference stages for the SEA-ALL architecture. The components with bold pink outlines are fine-tuned during the adaptation phase. The purpose of training is to produce a prior. This prior is combined with few data during adaptation to solve a new task. This adapted model is then deployed in the final inference stage. model is expressed as: DISPLAYFORM1 where a table of speaker embedding vectors e s (Embedding in FIG0) is learned alongside the standard WaveNet parameters. These vectors capture salient voice characteristics across individual speakers, and provide a convenient mechanism for generalizing WaveNet to the few-shot adaptation setting in this paper. The linguistic features l and fundamental frequency values f 0 are both time-series with a lower sampling frequency than the waveform. Thus, to be used as local conditioning variables they are upsampled by a transposed convolutional network. During training, l and f 0 are extracted by signal processing methods from pairs of training utterance and transcript, and during testing, those values are predicted from text by existing models. In recent years, a large body of literature uses large datasets to train models to learn an input-output mapping that is then used for inference. In contrast, few-shot meta-learning introduces an additional step, adaptation. In this meta-learning setting, the purpose of training becomes to learn a prior. During adaptation, this prior is combined with few data to rapidly learn a new skill; in this case adapting to a new speakers' voice style. Finally, the new skill is deployed, which in this paper we are referring to as inference. These three stages -training, adaptation and inference -are illustrated in Figure 2.We present two multi-speaker WaveNet extensions for few-shot voice adaptation. First, we introduce a non-parametric model fine-tuning approach, which involves adapting either the speaker embeddings or all the model parameters using held-aside demonstration data. Second, and for comparison purposes, we use a parametric approach whereby an auxiliary network is trained to predict the embedding vector of a new speaker using the demonstration data. Inspired by few-shot learning we first pre-train a multi-speaker conditional WaveNet model on a large and diverse dataset, as described in Section 2. Subsequently, we fine-tune the model parameters by retraining with respect to held-aside adaptation data. Training this WaveNet model to maximize the conditional log-likelihood of the generated audio jointly optimizes both the set of speaker parameters {e s} and the shared WaveNet core parameters w. Next, we extend this method to a new speaker by extracting the l and f 0 features from their adaptation data and randomly initializing a new embedding vector e. We then optimize e such that the demonstration waveforms, {x DISPLAYFORM0 0,demo)}, are likely under the model with w fixed (SEA-EMB): DISPLAYFORM1 Alternatively, all of the model parameters may be additionally fine-tuned (SEA-ALL): DISPLAYFORM2 0,demo; e, w).Both methods are non-parametric approaches to few-shot voice adaptation as the number of embedding vectors scales with the number of speakers. However, the training processes are slightly different. Because the SEA-EMB method optimizes only a low-dimensional vector, it is far less prone to overfitting, and we are therefore able to retrain the model to convergence even with mere seconds of adaptation data. By contrast, the SEA-ALL has many more parameters that might overfit to the adaptation data. We therefore hold out 10% of our demonstration data for calculating a standard early termination criterion. We also initialize e with the optimal value from the SEA-EMB method, and we find this initialization significantly improves the generalization performance even with a few seconds of adaptation data. In contrast to the non-parametric approach, whereby a different embedding vector is fitted for each speaker, one can train an auxiliary encoder network to predict an embedding vector for a new speaker given their demonstration data. Specifically, we model: DISPLAYFORM0 where for each training example, we include a randomly selected demonstration utterance from that speaker in addition to the regular conditioning inputs. The full WaveNet model and the encoder network e(·) are trained together from scratch. We refer the reader to the Appendix for further architectural details. This approach (SEA-ENC) exhibits the advantage of being trained in a transcriptindependent setting given only the input waveform, e(x demo), and requires negligible computation at adaptation time. However, the learned encoder can also introduce bias when fitting an embedding due to its limited network capacity. As an example, demonstrated a typical scenario whereby speaker identity information can be very quickly extracted with deep models from audio signals. Nonetheless, that the model is less capable of effectively leveraging additional training than approaches based on statistical methods. The linguistic features and fundamental frequencies which are used as inputs contain information specific to an individual speaker. As an example, the average voice pitch in the fundamental frequency sequence is highly speaker-dependent. Instead, we would like these features to be as speaker-independent as possible such that identity is modeled via global conditioning on the speaker embedding. To achieve this, we normalize the fundamental frequency values to have zero mean and unit variance separately for each speaker during training, denoted asf 0: DISPLAYFORM0 As mentioned earlier, at test time, we use an existing model to predict (l,f 0). Few-shot learning to build models, where one can rapidly learn using only a small amount of available data, is one of the most important open challenges in machine learning. Recent studies have attempted to address the problem of few-shot learning by using deep neural networks, and they have shown promising on classification tasks in vision BID11 BID12 and language. Few-shot learning can also be leveraged in reinforcement learning, such as by imitating human Atari gameplay from a single recorded action sequence BID14 or online video BID15.Meta-learning offers a sound framework for addressing few-shot learning. Here, an expensive learning process in machines with the ability to learn rapidly from few data. Meta-learning has a long history BID16 BID17, and recent studies include efforts to learn optimization processes BID18 that have been shown to extend naturally to the few-shot setting BID20 ). An alternative approach is model-agnostic meta learning (MAML) BID21, which differs by using a fixed optimizer and learning a set of base parameters that can be adapted to minimize any task loss by few steps of gradient descent. This method has shown promise in robotics BID22 BID23.In generative modeling, few-shot learning has been addressed from several perspectives, including matching networks BID24 and variable inference for memory addressing BID25. BID26 developed a sequential generative model that extended the Deep Recurrent Attention Writer (DRAW) model BID27, and BID28 extended PixelCNN with neural attention for few-shot auto-regressive density modeling. BID30 presented a gated linear model able to model complex densities from a single pass of a limited dataset. Early attempts of few-shot adaptation involved the attention models of BID28 and MAML BID21 ), but we found both of these strategies failed to learn informative speaker embedding in our preliminary experiments. There is growing interest in developing neural TTS models that can be trained end-to-end without the need for hand-crafted representations. In this study we focus on extending the autoregressive WaveNet model BID6 to the few-shot learning setting to adapt to speakers that were not presented at training time. Other recent neural TTS models include Tacotron 2 (building on) which uses WaveNet as a vocoder to invert mel-spectrograms generated by an attentive sequence-to-sequence model. DeepVoice 2 ) (building on BID34) introduced a multi-speaker variation of Tacotron that learns a low-dimensional embedding for each speaker, which was further extended in DeepVoice 3 to a 2,400 multi-speaker scenario. Unlike WaveNet and DeepVoice, the Char2Wav BID36 and VoiceLoop ) models produce World Vocoder Features BID38 instead of generating raw audio signals. Although many of these systems have produced high-quality samples for speakers present in the training set, generalizing to new speakers given only a few seconds of audio remains a challenge. There have been several concurrent works to address this few-shot learning problem. The VoiceLoop model introduced a novel memory-based architecture that was extended by to few-shot voice style adaptation, by introducing an auxiliary fitting network that predicts the embedding of a new speaker. BID40 extended the Tacotron model for one-shot speaker adaptation by conditioning on a speaker embedding vector extracted from a pretrained speaker identity model of BID8. The most similar approached to our work was proposed by for the DeepVoice 3 model. They considered both predicting the embedding with an encoding network and fitting the embedding based on a small amount of adaptation data, but the adaptation was applied to a prediction model for mel-spectrograms with a fixed vocoder. In this section, we evaluate the quality of samples of SEA-ALL, SEA-EMB and SEA-ENC. We first measure the naturalness of the generated utterances using the standard Mean Opinion Score (MOS) procedure. Then, we evaluate the similarity of generated and real samples using the subjective MOS test and objectively using a speaker verification system BID8. Finally, we study these varying the size of the adaptation dataset. We train a WaveNet model for each of our three methods using the same dataset, which combines the high-quality LibriSpeech audiobook corpus BID42 3.61 ± 0.06 3.56 ± 0.06 3.65 ± 0.06 3.58 ± 0.06 Table 1: Naturalness of the adapted voices using a 5-scale MOS score (higher is better) with 95% confidence interval on the LibriSpeech and VCTK held-out adaptation datasets. Numbers in bold are the best few-shot learning on each dataset without statistically significant difference. van den was trained with 24-hour production quality data, Our few-shot model performance is evaluated using two hold-out datasets. First, the LibriSpeech test corpus consists of 39 speakers, with an average of approximately 52 utterances and 5 minutes of audio per speaker. For every test speaker, we randomly split their demonstration utterances into an adaptation set for adapting our WaveNet models and a test set for evaluation. The subset of utterances used for early termination in Section 3.1 is chosen from the adaptation set. There are about 4.2 utterances on average per speaker in the test set and the rest in the adaptation set. Second, we consider a subset of the CSTR VCTK corpus BID43 consisting of 21 American English speakers, with approximately 368 utterances and 12 minutes of audio per speaker. We also apply the adaptation/test split with 10 utterances per speaker for test. We emphasize that no data from VCTK was presented to the model at training time. Since our underlying WaveNet model was trained on data largely from LibriSpeech (which was recorded under noisier conditions than VCTK), one might expect that the generated samples on the VCTK dataset contain characteristic artifacts that make generated samples easier to distinguish from real utterances. However, our evaluation using VCTK indicates that our model generalizes effectively and that such artifacts are not detectable. Synthetic utterances are provided on our demo webpage 1.It is worth mentioning, that SEA-ENC requires no adaptation time. Where for SEA-EMB, it takes 5 ∼ 10k optimizing steps to fit the embedding vector, and an additional 100 ∼ 200 steps to fine-tune the entire model using early stopping for SEA-ALL. We measure the quality of the generated samples by conducting a MOS test, whereby subjects are asked to rate the naturalness of generated utterances on a five-point Likert Scale (1: Bad, 2: Poor, 3: Fair, 4: Good, 5: Excellent). Furthermore, we compare with other published few-shot TTS systems systems, that were developed in parallel to this work. However, the literature uses varying combinations of training data and evaluation splits making comparison difficult. The presented are from the closest experimental setups to ours. Table 1 presents MOS for the adaptation models compared to real utterances. Two different adaptation dataset sizes are considered; T = 10 seconds, and T ≤ 5 minutes for LibriSpeech (T ≤ 10 minutes for VCTK). For reference on 16 kHz data, WaveNet trained on a 24-hour production quality speech dataset (van den) achieves a score of 4.21, while for LibriSpeech our best few-shot model attains an MOS score of 4.13 using only 5 minutes of data given a pre-trained multi-speaker model. We note that both fine-tuning models produce overall "good" samples for both the LibriSpeech and VCTK test sets, with SEA-ALL outperforming SEA-EMB in all cases. SEA-ALL is on par 3.41 ± 0.10 3.75 ± 0.09 3.51 ± 0.10 3.97 ± 0.09 SEA-EMB (ours)3.42 ± 0.10 3.56 ± 0.10 3.07 ± 0.10 3.18 ± 0.10 SEA-ENC (ours)2.47 ± 0.09 2.59 ± 0.09 2.07 ± 0.08 2.19 ± 0.09 Table 2: Voice similarity of generated voices using a 5-scale MOS score (higher is better) with 95% confidence interval on the LibriSpeech and VCTK held-out adaptation datasets.with the state-of-the-art performance on both datasets. The addition of extra adaptation data beyond 10 seconds of audio helps performance on LibriSpeech but not VCTK, and the gap between our best model and the real utterance is also wider on VCTK, possibly due to the different recording conditions. Beside naturalness, we also measure the similarity of the generated and real voices. The quality of similarity is the main evaluation metric for the voice adaptation problem. We first follow the experiment setup of BID40 to run a MOS test for a subjective assessment and then use a speaker verification model for objective evaluation in the next section. In every trial of this test a subject is presented with a pair of utterances consisting of a real utterance and another real or generated utterance from the same speaker, and is asked to rate the similarity in voice identity using a five-scale score (1: Not at all similar, 2: Slightly similar, 3: Moderately similar, 4: Very similar, 5: Extremely similar). Table 2 shows the MOS for real utterances and all the adaptation models under two adaptation data time settings on both datasets. Again, the SEA-ALL model outperforms the other two models, and the improvement over SEA-EMB scales with the amount of adaptation data. Particularly, the learned voices on the VCTK dataset achieve an average score of 3.97, demonstrating the generalization performance on a different dataset. As a rough comparisson, because of varying training setups, the state of the art system of BID40 achieves scores of 3.03 for LibriSpeech and 2.77 for VCTK when trained on LibriSpeech. Their model computes the embedding based on the d-vector, similar to our SEA-ENC approach, and performs competitively for the one-shot learning setting, but its performance saturates with 5 seconds of adaptation data, as explained in Section 3.2. We note the gap of similarity scores between SEA-ALL and real utterances, which suggests that although the generated samples sound similar to the target speakers, humans can still tell the difference from real utterances. We also apply the state-of-the-art text independent speaker verification (TI-SV) model of BID8 to objectively assess whether the generated samples preserve the acoustic features of the speakers. We calculate the TI-SV d-vector embeddings for generated and real voices. In Figure 3, we visualize the 2-dimensional projection of the d-vectors for a SEA-ALL model trained on T ≤ 5 minutes of data on the LibriSpeech dataset, and T ≤ 10 minutes on VCTK. There are clear clusters on both datasets, with a strikingly large inter-cluster distance and low intra-cluster separation. This shows both an ease of correctly identifying the speaker associated with a given generated utterance, and the difficulty in differentiating real from synthetic samples. A similar figure is presented in BID40, but there the generated and real samples do not overlap. This indicates that the method presented in this paper generates voices that are more indistinguishable from real ones, when measured with the same verification system. In the following subsections, we further analyze these . BID8. The utterances were generated using T ≤ 5 and T ≤ 10 minute samples from LibriSpeech and VCTK respectively. EER is marked with a dot. We first quantify whether generated utterances are attributed to the correct speaker. Following common practice in speaker verification BID8, we select the hold-out test set of real utterances from test speakers as the enrollment set and compute the centroid of the d-vectors for each speaker c i. We then use the adaptation set of test speakers as the verification set. For every verification utterance, we compute the cosine similarity between its d-vector v and a randomly chosen centroid c i. The utterance is accepted as one from speaker i if the similarity is exceeds a given threshold. We repeat the experiments with the same enrollment set and replace the verification set with samples generated by each adaptation method under different data size settings. Figure 5: Cosine similarity of real and generated utterances to the real enrollment set. In our setup we fix the enrollment set together with the speaker verification model from BID8, and study the performance of different verification sets that are either from real utterances or generated by a TTS system. TAB4 lists the equal error rate (EER) of the verification model with real and generated verification utterances, and Figure 4 shows the detection error trade-off (DET) curves for a more thorough inspection. Figure 4 only shows the adaptation models with the maximum data size setting (T ≤ 5 minutes for LibriSpeech and ≤ 10 minutes for VCTK). The for other data sizes are provided in Appendix B.We find that SEA-ALL outperforms the other two approaches, and the error rate decreases clearly with the size of demonstration data. Noticeably, the EER of SEA-ALL is even lower than the real utterance on the LibriSpeech dataset with sufficient adaptation data. A possible explanation is that the generated samples might be concentrated closer to the centroid of a speaker's embeddings than real speech with larger variance across utterances. Our SEA-EMB model performs better than SEA-ENC. Additionally, the benefit of more demonstration data is less significant than for SEA-ALL in both of these models. In this section, we compare the generated samples and the real utterances of the speaker being imitated. Figure 5 shows the box-plot of the cosine similarity between the embedding centroids of test speakers' enrollment set and real utterances from the same speaker, real utterances from a different speaker, and generated utterances adapted to the same speaker. Consistent with the observations from the previous subsection, SEA-ALL performs best. We further consider an adversarial scenario for speaker verification. In contrast to the previous standard speaker verification setup where we now select a verification utterance with either a real utterance from the same speaker or a synthetic sample from a model adapted to the same speaker. Under this setup, the speaker verification system is challenged by synthetic samples and acts as a classifier for real versus generated utterances. The ROC curve of this setup is shown in FIG3 and the models are using the maximum data size setting. Other data size settings can be found in Appendix C. If the generated samples are indistinguishable from real utterances, the ROC curve approaches the diagonal line (that is, the verification system fails to separate real and generated voices). Importantly, SEA-ALL manages to confuse the verification system especially for the VCTK dataset where the ROC curve is almost inline with the diagonal line with an AUC of 0.56. This paper studied three variants of meta-learning for sample efficient adaptive TTS. The adaptation method that fine-tunes the entire model, with the speaker embedding vector first optimized, shows impressive performance even with only 10 seconds of audio from new speakers. When adapted with a few minutes of data, our model matches the state-of-the-art performance in sample naturalness. Moreover, it outperforms other recent works in matching the new speaker's voice. We also demon-: ROC curve for real versus generated utterance detection. The utterances were generated using models with 5 and 10 minutes of training data per speaker from LibriSpeech and VCTK respectively. Lower curve indicate that the verification system is having a harder time distinguishing real from generated samples.strated that the generated samples achieved a similar level of voice similarity to real utterances from the same speaker, when measured by a text independent speaker verification model. Our paper considers the adaptation to new voices with clean, high-quality training data collected in a controlled environment. The few-shot learning of voices with noisy data is beyond the scope of this paper and remains a challenging open research problem. A requirement for less training data to adapt the model, however, increases the potential for both beneficial and harmful applications of text-to-speech technologies such as the creation of synthesized media. While the requirements for this particular model (including the high-quality training data collected in a controlled environment and equally high quality data from the speakers to which we adapt, as described in Section 5.1) present barriers to misuse, more research must be conducted to mitigate and detect instances of misuse of text-to-speech technologies in general. Our encoding network is illustrated as the summation of two sub-network outputs in FIG4. The first subnetwork is a pre-trained speaker verification model (TI-SV) BID8, comprising 3 LSTM layers and a single linear layer. This model maps a waveform sequence of arbitrary length to a fixed 256-dimensional d-vector with a sliding window, and is trained from approximately 36M utterances from 18K speakers extracted from anonymized voice search logs. On top of this we add a shallow MLP to project the output d-vector to the speaker embedding space. The second sub-network comprises 16 1-D convolutional layers. This network reduces the temporal resolution to 256 ms per frame (for 16 kHz audio), then averages across time and projects into the speaker embedding space. The purpose of this network is to extract residual speaker information present in the demonstration waveforms but not captured by the pre-trained TI-SV model. Here we provide the DET curves of speaker verification problem for models with different training data sizes in addition to those shown in Section 5.4.1. Figure 8: Detection error trade-off (DET) curve for speaker verification, using the TI-SV speaker verification model BID8. The utterances were generated using 1 minute or 10 seconds of utterance from LibriSpeech and VCTK. EER is marked with a dot. We provide the ROC curves of the speaker verification problem with adversarial examples from adaptation models with different training data sizes in addition to those shown in Section 5.4.2.: ROC curve for real vs. generated utterance detection. The utterances were generated using 1 minute or 10 seconds of utterance from LibriSpeech and VCTK. Lower curve suggests harder to distinguish real from generated samples., a positive would be Sample efficient algorithms to adapt a text-to-speech model to a new voice style with the state-of-the-art performance. & a negative would be We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.

Question: End-to-end acoustic-to-word speech recognition models have recently gained popularity because they are easy to train, scale well to large amounts of training data, and do not require a lexicon. In addition, word models may also be easier to integrate with downstream tasks such as spoken language understanding, because inference (search) is much simplified compared to phoneme, character or any other sort of sub-word units. In this paper, we describe methods to construct contextual acoustic word embeddings directly from a supervised sequence-to-sequence acoustic-to-word speech recognition model using the learned attention distribution. On a suite of 16 standard sentence evaluation tasks, our embeddings show competitive performance against a word2vec model trained on the speech transcriptions. In addition, we evaluate these embeddings on a spoken language understanding task and observe that our embeddings match the performance of text-based embeddings in a pipeline of first performing speech recognition and then constructing word embeddings from transcriptions. The task of learning fixed-size representations for variable length data like words or sentences, either text or speech-based, is an interesting problem and a focus of much current research. In the natural language processing community, methods like word2vec BID0, GLoVE BID1, CoVe BID2 and ELMo BID3 have become increasingly popular, due to their utility in several natural language processing tasks. Similar research has progressed in the speech recognition community, where however the input is a sequence of short-term audio features, rather than words or characters. Therefore, the variability in speakers, acoustics or microphones for different occurrences of the same word or sentence adds to the challenge. Prior work towards the problem of learning word representations from variable length acoustic frames involved either providing word boundaries to align speech and text BID4, or chunking ("chopping" or "padding") input speech into fixed-length segments that usually span only one word BID5 BID6 BID7 BID8. Since these techniques learn acoustic word embeddings from audio fragment and word pairs obtained via a given segmentation of the audio data, they ignore the specific audio context associated with a particular word. So the ing word embeddings do not capture the contextual dependencies in speech. In contrast, our work constructs individual acoustic word embeddings grounded in utterance-level acoustics. In this paper, we present different methods of obtaining acoustic word embeddings from an attention-based sequence-to-sequence * Equal contribution model BID9 BID10 BID11 trained for direct Acoustic-to-Word (A2W) speech recognition BID12. Using this model, we jointly learn to automatically segment and classify input speech into individual words, hence getting rid of the problem of chunking or requiring pre-defined word boundaries. As our A2W model is trained at the utterance level, we show that we can not only learn acoustic word embeddings, but also learn them in the proper context of their containing sentence. We also evaluate our contextual acoustic word embeddings on a spoken language understanding task, demonstrating that they can be useful in non-transcription downstream tasks. Our main contributions in this paper are the following: 1. We demonstrate the usability of attention not only for aligning words to acoustic frames without any forced alignment but also for constructing Contextual Acoustic Word Embeddings (CAWE). 2. We demonstrate that our methods to construct word representations (CAWE) directly from a speech recognition model are highly competitive with the text-based word2vec embeddings BID0, as evaluated on 16 standard sentence evaluation benchmarks. 3. We demonstrate the utility of CAWE on a speech-based downstream task of Spoken Language Understanding showing that pretrained speech models could be used for transfer learning similar to VGG in vision BID13 or CoVe in natural language understanding BID2. A2W modeling has been largely pursued using Connectionist Temporal Classification (CTC) models BID14 and Sequence-to-Sequence (S2S) models BID9. Prior work shows the need for large amounts of training data for these models (thousands of hours of speech) with large word vocabularies of frequently occurring words BID15 BID16 BID17 BID18 BID19. Progress in the field showed the possibility of training these models with smaller amount of data (300 hours Switchboard corpus BID20) but restricting the vocabulary to words occurring atleast 5 or 10 times BID21 BID22. The solutions to generate out-of-vocabulary words have revolved around backing off to smaller units like characters or sub-words BID21 BID16 BID17 BID22 BID19. While this solves the problem of rare word generation, the models are no longer pure-word models. present one of the first S2S models for pure-word large vocabulary A2W recognition with the 300 hour Switchboard corpus with a vocabulary of about 30,000 words. BID24 BID12 build upon their work and improve the training of these models for the large vocabulary task. BID12 is one of our previous works where we show that the direct A2W model is also able to automatically learn word boundaries without any supervision and is the current best pure-word S2S model. We use the same model in this work and expand it towards learning acoustic embeddings. BID4 BID5 BID7 BID6 BID25 BID8 all explore ways to learn acoustic word embeddings. All above methods except BID6 use unsupervised learning based methods to obtain these embeddings where they do not use the transcripts or do not perform speech recognition. BID6 use a supervised Convolutional Neural Network based speech recognition model but with short speech frames as input that usually correspond to a single word. This is the common practice in most prior work that simplifies training but prevents the models to scale to learn contextual word embeddings grounded in utterance level acoustics. BID4 propose an unsupervised method to learn speech embeddings using a fixed context of words in the past and future. The drawbacks of their method are the fixed context and need for forced alignment between speech and words for training. Learning text-based word embeddings is also a rich area of research with well established techniques such as BID0 BID1. Research has further progressed into learning contextualized word embeddings BID2 BID3 that are useful in many text-based downstream tasks BID26. BID2 learns contextual word embeddings from a fully trained machine translation model and depict re-use of their encoder in other downstream tasks. Our work ties A2W speech recognition model with learning contextual word embeddings from speech. Our S2S model is similar in structure to the Listen, Attend and Spell model BID10 which consists of 3 components: the encoder network, a decoder network and an attention model. The encoder maps the input acoustic features vectors a = (a1, a2, ..., aT) where ai ∈ R d, into a sequence of higher-level features h = (h1, h2, ..., h T). The encoder is a pyramidal (sub-sampling) multi-layer bi-directional Long Short Term Memory (BLSTM) network. The decoder network is also an LSTM network that learns to model the output distribution over the next target conditioned on sequence of previous predictions i.e. P (y l |y * DISPLAYFORM0) is the ground-truth label sequence. In this work, y * i ∈ U is from a word vocabulary. This decoder generates targets y from h using an attention mechanism. We use the location-aware attention mechanism BID11 that enforces monotonicity in the alignments by applying a convolution across time to the attention of previous time step. This convolved attention feature is used for calculating the attention for the current time step which leads to a peaky distribution BID11 BID12. Our model follows the same experimental setup and model hyper-parameters as the word-based models described in our previous work BID12 with the difference of learning 300 dimensional acoustic feature vectors instead of 320 dimensional. We now describe our method to obtain the acoustic word embeddings from the end-to-end trained speech recognition system described in Section 3. The model is as shown in Figure 1 where the embeddings are constructed using the hidden representations obtained from the encoder and the attention weights from the decoder. Our method of constructing "contextual" acoustic word embeddings is similar to a method proposed for text embeddings, CoVe BID2. The main challenge that separates our method from CoVe BID2 in learning embeddings from a supervised task, is the problem of alignment between input speech and output words. We use the location-aware attention mechanism that has the property to assign higher probability to certain frames leading to a peaky attention distribution. We exploit this property of location-aware attention in an A2W model to automatically segment continuous speech into words as shown in our previous work BID12, and then use this segmentation to obtain word embeddings. In the next two subsections, we formalize this process of constructing contextual acoustic word embeddings. Intuitively, attention weights on the acoustic frames hidden representations reflect their importance in classifying a particular word. They thereby provide a correspondence between the frame and the word within a given acoustic context. We can thus construct word representations by weighing the hidden representations of these acoustic frames in terms of their importance to the word i.e. the attention weight. We show this in the Figure 1 wherein the hidden representations and their attention weights are colored according to their correspondence with a particular word. Given that aj represents the acoustic frame j, let encoder(aj) represent the higher-level features obtained for the frame aj (i.e. encoder(aj) = h = (h1, h2, ..., h T), as explained in Section 3). Then, for the i th word wi our model first obtains the mappings of wi to acoustic frames aK where K is the set such that ∀k ∈ K k = arg max j (attention(aj)) over all utterances U containing the word wi in the training set. Below we describe three different ways of using attention to obtain acoustic word embeddings for a word wi (here, n(K) represents the cardinality of the set K): DISPLAYFORM0 Therefore, unweighted Average (U-AVG, Equation 1) is just the unweighted combination of all the hidden representations of acoustic frames mapped to a particular word. Attention weighted Average (CAWE-W, Equation 2) is the weighted average of the hidden representations of all acoustic frames using the attention weights for a given word. Finally, maximum attention (CAWE-M, Equation 3) is the hidden representation of the acoustic frame with the highest attention score for a given word across all utterances in the training data. We call the attention-weighted average and the maximum attention based techniques as Contextual Acoustic Word Embeddings (CAWE) since they are contextual owing to the use of attention scores (over all acoustic frames for a given word). We use a commonly used speech recognition setup, the 300 hour Switchboard corpus (LDC97S62) BID20 which consists of 2,430 twosided telephonic conversations between 500 different speakers and contains 3 million words of text. Our second dataset is a 300 hour subset of the How2 BID27 dataset of instructional videos, which contains planned, but free speech, often outdoor and recorded with distant microphones, as opposed to the indoor, telephony, conversational speech of Switchboard. There are 13,662 videos with a total of 3.5 million words in this corpus. The A2W obtains a word error rate of 22.2% on Switchboard and 36.6% on CallHome set from the Switchboard Eval2000 test set and 24.3% on dev5 test set of How2. Datasets for Downstream Tasks: We evaluate our embeddings by using them as features for 16 benchmark sentence evaluation tasks that cover Semantic Textual Similarity (STS 2012-2016 and STS B), classification: Movie Review (MR), product review (CJ), sentiment analysis (SST, SST-FG), question type (TREC), Subjectivity/Objectivity (SUBJ), and opinion polarity (MPQA), entailment and semantic relatedness using the SICK dataset for SICK-E (entailment) and SICK-R (relatedness) and paraphrase detection (MRPC). The STS and SICK-R tasks measure Spearman's coefficient of correlation between embedding based similarity and human scores, hence the scores range from [−1, 1] where higher number denotes high correlation. All the remaining tasks are measured on test classification accuracies. We use the SentEval toolkit BID26 to evaluate. Training Details: In all downstream evaluations involving classification tasks, we have used a simple logistic regression for classification since a better representation should lead to better scores without using complicated models (hence abstracting away model complexities from our evaluations). This also means that we can use the concatenation of CAWE and CBOW as features to the logistic regression model without adding tunable embedding parameters. Discussion: From the in TAB1 we see that CAWE-M outperforms U-AVG by 34% and 13% and CAWE-W by 33.9% and 12% on Switchboard and How2 datasets respectively in terms of average performance on STS tasks and leads to better or slightly worse performance on the classification tasks. We observe that CAWE-W usually performs worse than CAWE-M which could be attributed to a noisy estimation of the word embeddings on the account of taking even the less confident attention scores while constructing the embedding. In contrast, CAWE-M is constructed using the most confident attention score obtained over all the occurrences of the acoustic frames corresponding to a particular word. We also observe that U-AVG performs worse than CAWE-W on STS and SICK-R tasks since it is constructed using an even noisier process in which all encoder hidden representations are weighted equally irrespective of their attention scores. Datasets for Downstream Tasks: The datasets are the same as described in Section 5.1.Training Details: In all the following comparisons, we compare embeddings obtained only from the training set of the speech recognition model, while the text-based word embeddings are obtained by training Continuous Bag-of-Words (CBOW) word2vec model on all the transcripts (train, validation and test). This was done to ensure a fair comparison between our supervised technique and the unsupervised word2vec method. This naturally leads to a smaller vocabulary for CAWE. Further, one of the drawbacks of A2W speech recognition model is that it fails to capture entire vocabulary, recognizing only 3044 words out of 29874 (out of which 18800 words occur less than 5 times) and 4287 out of 14242 total vocabulary for Switchboard and How2 respectively. Despite this fact, the performance of CAWE is very competitive with word2vec CBOW which does not TAB2, we see that our embeddings perform as well as the text-embeddings. Evaluations using CAWE-M extracted from Switchboard based training show that the acoustic embeddings when concatenated with the text embeddings outperform the word2vec embeddings on 10 out of 16 tasks. This concatenated embedding shows that we add more information with CAWE-M that improves the CBOW embedding as well. The gains are more prominent in Switchboard as compared to the How2 dataset since How2 is planned instructional speech whereas Switchboard is spontaneous conversational speech (thereby making the How2 characteristics closer to text leading to a stronger CBOW model). Dataset: In addition to generic sentence-level evaluations, we also evaluate CAWE on the widely used ATIS dataset BID28 for Spoken Language Understanding (SLU). ATIS dataset is comprised of spoken language queries for airline reservations that have intent and named entities. Hence, it is similar in domain to Switchboard, making it a useful test bed for evaluating CAWE on a speech-based downstream evaluation task. Training Details: For this task, our model is similar to the simple Recurrent Neural Network (RNN) based model architecture as investigated in BID29. Our architecture is comprised of an embedding layer, a single layer RNN-variant (Simple RNN, Gated Recurrent Unit (GRU)) along with a dense layer and softmax. In each instance, we train our model for 10 epochs with RMSProp (learning rate 0.001). We train each model 3 times with different seed values and report average performance. Discussion: BID29 concluded that text-based word embeddings trained on large text corpora consistently lead to better performance on the ATIS dataset. We demonstrate that direct speech-based word embeddings could lead to matching performance when compared to text-based word embeddings in this speech-based downstream task, thus highlighting the utility of our speech based embeddings. Specifically, we compare the test scores obtained by initializing the model with CAWE-M, CAWE-W and CBOW embeddings and fine-tuning them based on the task. We present a method to learn contextual acoustic word embeddings from a sequence-to-sequence acoustic-to-word speech recognition model that learns to jointly segment and classify speech. We analyze the role of attention in constructing contextual acoustic word embeddings, and find our acoustic embeddings to be highly competitive with word2vec (CBOW) text embeddings. We discuss two variants of such contextual acoustic word embeddings which outperform the simple unweighted average method by upto 34% on semantic textual similarity tasks. The embeddings also matched the performance of text-based embeddings in spoken language understanding, showing the use of this model as a pre-trained model for other speech-based downstream tasks. We surmise that contextual audio embeddings will generalize and improve downstream tasks in a way that is similar to their text counterparts, despite the additional complexity presented by noisy audio input. In the future, we will explore ways to scale our model to larger corpora, larger vocabularies and compare with non-contextual acoustic word embedding methods. This work was supported by the Center for Machine Learning and Health (CMLH) at Carnegie Mellon University and by Facebook.

Represent the short summary for finding the corresponding scientific article (long)

Methods to learn contextual acoustic word embeddings from an end-to-end speech recognition model that perform competitively with text-based word embeddings.

We provide insightful understanding of sequence-labeling NER and propose to use two types of cross structures, both of which bring theoretical and empirical improvements.