Question: Represent the article for finding a single-sentence TLDR

Question: Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales which time-domain models have yet to achieve. We demonstrate that our model captures longer-range dependencies than time-domain models such as WaveNet across a diverse set of unconditional generation tasks, including single-speaker speech generation, multi-speaker speech generation, and music generation. Audio waveforms have complex structure at drastically varying timescales, which presents a challenge for generative models. Local structure must be captured to produce high-fidelity audio, while longrange dependencies spanning tens of thousands of timesteps must be captured to generate audio which is globally consistent. Existing generative models of waveforms such as WaveNet (van den a) and SampleRNN are well-adapted to model local dependencies, but as these models typically only backpropagate through a fraction of a second, they are unable to capture high-level structure that emerges on the scale of several seconds. We introduce a generative model for audio which captures longer-range dependencies than existing end-to-end models. We primarily achieve this by modelling 2D time-frequency representations such as spectrograms rather than 1D time-domain waveforms (Figure 1). The temporal axis of a spectrogram is orders of magnitude more compact than that of a waveform, meaning dependencies that span tens of thousands of timesteps in waveforms only span hundreds of timesteps in spectrograms. In practice, this enables our spectrogram models to generate unconditional speech and music samples with consistency over multiple seconds whereas time-domain models must be conditioned on intermediate features to capture structure at similar timescales. Modelling spectrograms can simplify the task of capturing global structure, but can weaken a model's ability to capture local characteristics that correlate with audio fidelity. Producing high-fidelity audio has been challenging for existing spectrogram models, which we attribute to the lossy nature of spectrograms and oversmoothing artifacts which from insufficiently expressive models. To reduce information loss, we model high-resolution spectrograms which have the same dimensionality as their corresponding time-domain signals. To limit oversmoothing, we use a highly expressive autoregressive model which factorizes the distribution over both the time and frequency dimensions. Modelling both fine-grained details and high-level structure in high-dimensional distributions is known to be challenging for autoregressive models. To capture both local and global structure in spectrograms with hundreds of thousands of dimensions, we employ a multiscale approach which generates spectrograms in a coarse-to-fine manner. A low-resolution, subsampled spectrogram that captures high-level structure is generated initially, followed by an iterative upsampling procedure that adds high-resolution details. (1x, 5x, 25x, 125x) Figure 1: Spectrogram and waveform representations of the same 4 second audio signal. The waveform spans nearly 100,000 timesteps whereas the temporal axis of the spectrogram spans roughly 400. Complex structure is nested within the temporal axis of the waveform at various timescales, whereas the spectrogram has structure which is smoothly spread across the time-frequency plane. Combining these representational and modelling techniques yields a highly expressive and broadly applicable generative model of audio. Our contributions are are as follows: • We introduce MelNet, a generative model for spectrograms which couples a fine-grained autoregressive model and a multiscale generation procedure to jointly capture local and global structure. • We show that MelNet is able to model longer-range dependencies than existing time-domain models. Additionally, we include an ablation to demonstrate that multiscale modelling is essential for modelling long-range dependencies. • We demonstrate that MelNet is broadly applicable to a variety of audio generation tasks, including unconditional speech and music generation. Furthermore, MelNet is able to model highly multimodal data such as multi-speaker and multilingual speech. We briefly present regarding spectral representations of audio. Audio is represented digitally as a one-dimensional, discrete-time signal y = (y 1, . . ., y n). Existing generative models for audio have predominantly focused on modelling these time-domain signals directly. We instead model spectrograms, which are two-dimensional time-frequency representations which contain information about how the frequency content of an audio signal varies through time. Spectrograms are computed by taking the squared magnitude of the short-time Fourier transform (STFT) of a time-domain signal, i.e. x = STFT(y) 2. The value of x ij (referred to as amplitude or energy) corresponds to the squared magnitude of the jth element of the frequency response at timestep i. Each slice x i, * is referred to as a frame. We assume a time-major ordering, but following convention, all figures are displayed transposed and with the frequency axis inverted. Time-frequency representations such as spectrograms highlight how the tones and pitches within an audio signal vary through time. Such representations are closely aligned with how humans perceive audio. To further align these representations with human perception, we convert the frequency axis to the Mel scale and apply an elementwise logarithmic rescaling of the amplitudes. Roughly speaking, the Mel transformation aligns the frequency axis with human perception of pitch and the logarithmic rescaling aligns the amplitude axis with human perception of loudness. Spectrograms are lossy representations of their corresponding time-domain signals. The Mel transformation discards frequency information and the removal of the STFT phase discards temporal information. When recovering a time-domain signal from a spectrogram, this information loss manifests as distortion in the recovered signal. To minimize these artifacts and improve the fidelity of generated audio, we model high-resolution spectrograms. The temporal resolution of a spectrogram can be increased by decreasing the STFT hop size, and the frequency resolution can be increased by The context x <ij (grey) for the element x ij (black) is encoded using 4 RNNs. Three of these are used in the time-delayed stack to extract features from preceding frames. The fourth is used in the frequency-delayed stack to extract features from all preceding elements within the current frame. Each arrow denotes an individual RNN cell and arrows of the same color use shared parameters. increasing the number of Mel channels. Generated spectrograms are converted back to time-domain signals using classical spectrogram inversion algorithms. We experiment with both Griffin-Lim and a gradient-based inversion algorithm (Decorsière et al., 2015), and ultimately use the latter as it generally produced audio with fewer artifacts. We use an autoregressive model which factorizes the joint distribution over a spectrogram x as a product of conditional distributions. Given an ordering of the dimensions of x, we define the context x <ij as the elements of x that precede x ij. We default to a row-major ordering which proceeds through each frame x i, * from low to high frequency, before progressing to the next frame. The joint density is factorized as where θ ij parameterizes a univariate density over x ij. We model each factor distribution as a Gaussian mixture model with K components. Thus, θ ij consists of 3K parameters corresponding to means, and mixture coefficients {π ijk} K k=1. The ing factor distribution can then be expressed as Following the work on Mixture Density Networks and their application to autoregressive models , θ ij is modelled as the output of a neural network and computed as a function of the context x <ij. Precisely, for some network f with parameters ψ, we have θ ij = f (x <ij ; ψ). A maximum-likelihood estimate for the network parameters is computed by minimizing the negative log-likelihood via gradient descent. To ensure that the network output parameterizes a valid Gaussian mixture model, the network first computes unconstrained parameters {μ ijk,σ ijk,π ijk} K k=1 as a vectorθ ij ∈ R 3K, and enforces constraints on θ ij by applying the following transformations: These transformations ensure the standard deviations σ ijk are positive and the mixture coefficients π ijk sum to one. To model the distribution in an autoregressive manner, we design a network which computes the distribution over x ij as a function of the context x <ij. The network architecture draws inspiration from existing autoregressive models for images (; van den c; b; ; ;). In the same way that these models estimate a distribution pixel-by-pixel over the spatial dimensions of an image, our model estimates a distribution element-by-element over the time and frequency dimensions of a spectrogram. A noteworthy distinction is that spectrograms are not invariant to translation along the frequency axis, making 2D convolution less desirable than other 2D network primitives which do not assume invariance. Utilizing multidimensional recurrence instead of 2D convolution has been shown to be beneficial when modelling spectrograms in discriminative settings ), which motivates our use of an entirely recurrent architecture. Similar to Gated PixelCNN (van den b), the network has multiple stacks of computation. These stacks extract features from different segments of the input to collectively summarize the full context x <ij: • The time-delayed stack computes features which aggregate information from all previous frames x <i, *. • The frequency-delayed stack utilizes all preceding elements within a frame, x i,<j, as well as the outputs of the time-delayed stack, to summarize the full context x <ij. The stacks are connected at each layer of the network, meaning that the features generated by layer l of the time-delayed stack are used as input to layer l of the frequency-delayed stack. To facilitate the training of deeper networks, both stacks use residual connections . The outputs of the final layer of the frequency-delayed stack are used to compute the unconstrained parametersθ. The time-delayed stack utilizes multiple layers of multidimensional RNNs to extract features from x <i, *, the two-dimensional region consisting of all frames preceding x ij. Each multidimensional RNN is composed of three one-dimensional RNNs: one which runs forwards along the frequency axis, one which runs backwards along the frequency axis, and one which runs forwards along the time axis. Each RNN runs along each slice of a given axis, as shown in Figure 2. The output of each layer of the time-delayed stack is the concatenation of the three RNN hidden states. We denote the function computed at layer l of the time-delayed stack (three RNNs followed by concatenation) as F To ensure that h f ij [l] is computed using only elements in the context x <ij, the inputs to the frequencydelayed stack are shifted backwards one step along the frequency axis: h At the final layer, layer L, a linear map is applied to the output of the frequency-delayed stack to produce the unconstrained Gaussian mixture model parameters, i.e. To incorporate conditioning information into the model, conditioning features z are simply projected onto the input layer along with the inputs x: Reshaping, upsampling, and broadcasting can be used as necessary to ensure the conditioning features have the same time and frequency shape as the input spectrogram, e.g. a one-hot vector representation for speaker ID would first be broadcast along both the time and frequency axes. To improve audio fidelity, we generate high-resolution spectrograms which have the same dimensionality as their corresponding time-domain representations. Under this regime, a single training example has several hundreds of thousands of dimensions. Capturing global structure in such high-dimensional distributions is challenging for autoregressive models, which are biased towards capturing local dependencies. To counteract this, we utilize a multiscale approach which effectively permutes the autoregressive ordering so that a spectrogram is generated in a coarse-to-fine order. The elements of a spectrogram x are partitioned into G tiers x 1,..., x G, such that each successive tier contains higher-resolution information. We define x <g as the union of all tiers which precede x g, i.e. x <g = (x 1, . . ., x g−1). The distribution is factorized over tiers: and the distribution of each tier is further factorized element-by-element as described in Section 3. We explicitly include the parameterization by ψ = (ψ 1, . . ., ψ G) to indicate that each tier is modelled by a separate network. Figure 5: Schematic showing how tiers of the multiscale model are interleaved and used to condition the distribution for the subsequent tier. a) The initial tier is generated unconditionally. b) The second tier is generated conditionally given the the initial tier. c) The outputs of tiers 1 and 2 are interleaved along the frequency axis and used to condition the generation of tier 3. d) Tier 3 is interleaved along the time axis with all preceding tiers and used to condition the generation of tier 4. During training, the tiers are generated by recursively partitioning a spectrogram into alternating rows along either the time or frequency axis. We define a function split which partitions an input into even and odd rows along a given axis. The initial step of the recursion applies the split function to a spectrogram x, or equivalently x <G+1, so that the even-numbered rows are assigned to x G and the odd-numbered rows are assigned to x <G. Subsequent tiers are defined similarly in a recursive manner: At each step of the recursion, we model the distribution p(x g | x <g ; ψ g). The final step of the recursion models the unconditional distribution over the initial tier p(x 1 ; ψ 1). To model the conditional distribution p(x g | x <g ; ψ g), the network at each tier needs a mechanism to incorporate information from the preceding tiers x <g. To this end, we add a feature extraction network which computes features from x <g which are used condition the generation of x g. We use a multidimensional RNN consisting of four one-dimensional RNNs which run bidirectionally along slices of both axes of the context x <g. A layer of the feature extraction network is similar to a layer of the time-delayed stack, but since the feature extraction network is not causal, we include an RNN which runs backwards along the time axis and do not shift the inputs. The hidden states of the RNNs in the feature extraction network are used to condition the generation of x g. As each tier doubles the resolution, the features extracted from x <g have the same time and frequency shape as x g, allowing the conditioning mechanism described in section 4.3 to be used straightforwardly. To sample from the multiscale model we iteratively sample a value for x g conditioned on x <g using the learned distributions defined by the estimated network parametersψ = (ψ 1, . . .,ψ G). The initial tier, x 1, is generated unconditionally by sampling from p(x 1 ;ψ 1) and subsequent tiers are sampled from p(x g | x <g ;ψ g). At each tier, the sampled x g is interleaved with the context x <g: The interleave function is simply the inverse of the split function. Sampling terminates once a full spectrogram, x <G+1, has been generated. A spectrogram generated by a multiscale model is shown in Figure 4 and the sampling procedure is visualized schematically in Figure 5. To demonstrate the MelNet is broadly applicable as a generative model for audio, we train the model on a diverse set of audio generation tasks (single-speaker speech generation, multi-speaker speech generation, and music generation) using three publicly available datasets. Generated audio samples for each task are available on the accompanying web page https://audio-samples.github.io. We include samples generated using the priming and biasing procedures described by. Biasing lowers the temperature of the predictive distribution and priming seeds the model state with a given sequence of audio prior to sampling. Hyperparameters for all experiments are available in Appendix A. Speech and music have rich hierarchies of latent structure. Speech has complex linguistic structure (phonemes, words, syntax, semantics, etc.) and music has highly compositional musical structure (notes, chords, melody and rhythm, etc.). The presence of these latent structures in generated samples can be used as a proxy for how well a generative model has learned dependencies at various timescales. As such, a qualitative analysis of unconditional samples is an insightful method of evaluating generative models of audio. To facilitate such a qualitative evaluation, we train MelNet on each of the three unconditional generation tasks and include samples on the accompanying web page. For completeness, we briefly provide some of our own qualitative observations regarding the generated samples (Sections 6.1, 6.2, and 6.3). In addition to qualitative analysis, we conduct a human evaluation experiment to quantitatively compare how well WaveNet and MelNet capture high-level structure (Section 6.4). Lastly, we ablate the impact of the multiscale generation procedure on MelNet's ability model long-range dependencies (Section 6.5). To test MelNet's ability to model a single speaker in a controlled environment, we utilize the Blizzard 2013 dataset , which consists of audiobook narration performed in a highly animated manner by a professional speaker. We find that MelNet frequently generates samples that contain coherent words and phrases. Even when the model generates incoherent speech, the intonation, prosody, and speaking style remain consistent throughout the duration of the sample. Furthermore, the model learns to produce speech using a variety of character voices and learns to generate samples which contain elements of narration and dialogue. Biased samples tend to contain longer strings of comprehensible words but are read in a less expressive fashion. When primed with a real sequence of audio, MelNet is able to continue sampling speech which has consistent speaking style and intonation. Audiobook data is recorded in a highly controlled environment. To demonstrate MelNet's capacity to model distributions with significantly more variation, we utilize the VoxCeleb2 dataset . The VoxCeleb2 dataset consists of over 2,000 hours of speech data captured with real world noise including laughter, cross-talk, channel effects, music and other sounds. The dataset is also multilingual, with speech from speakers of 145 different nationalities, covering a wide range of accents, ages, ethnicities and languages. When trained on the VoxCeleb2 dataset, we find that MelNet is able to generate unconditional samples with significant variation in both speaker characteristics (accent, language, prosody, speaking style) as well as acoustic conditions ( noise and recording quality). While the generated speech is often not comprehensible, samples can often be identified as belonging to a specific language, indicating that the model has learned distinct modalities for different languages. Furthermore, it is difficult to distinguish real and fake samples which are spoken in foreign languages. For foreign languages, semantic structures are not understood by the listener and cannot be used to discriminate between real and fake. Consequently, the listener must rely largely on phonetic structure, which MelNet is able to realistically model. To show that MelNet can model audio modalities other than speech, we apply the model to the task of unconditional music generation. We utilize the MAESTRO dataset, which consists of over 172 hours of solo piano performances. The samples demonstrate that MelNet learns musical structures such as melody and harmony. Furthermore, generated samples often maintain consistent tempo and contain interesting variation in volume, timbre, and rhythm. Making quantitative comparisons with existing generative models such as WaveNet is difficult for various reasons and previous works have ultimately relied on largely empirical evaluations by the reader. To allow the reader to make these judgements for themselves, we provide samples from both WaveNet and MelNet for each of the tasks described in the previous sections. Furthermore, in an effort to provide quantitative metrics to support the claim that MelNet generates samples with improved long-range structure in comparison to WaveNet, we conduct a human experiment whereby participants are presented anonymized samples from both models and asked to select which sample exhibits longer-term structure. We resort to such evaluations since standard metrics for evaluation of generative models such as density estimates cannot be used to compare WaveNet and MelNet as that these models operate on different representations. The methodology for this experiment is as follows. For each of the three unconditional audio generation tasks, we generated 50 samples from WaveNet and 50 samples from MelNet. Participants were shown an anonymized, randomly-drawn sample from each model and instructed to "select the sample which has more coherent long-term structure." We collected 50 evaluations for each task. Results, shown in Table 1a, show that evaluators overwhelmingly agreed that samples generated by MelNet had more coherent long-range structure than samples from WaveNet across all tasks. In addition to comparing MelNet to an unconditional WaveNet model for music generation, we also compare to a two-stage Wave2Midi2Wave model which conditions WaveNet on MIDI generated by a separately-trained Music Transformer. The two-stage Wave2Midi2Wave model has the advantage of directly modelling labelled musical notes which distill much of the salient, high-level structure in music into a compact symbolic representation. Despite this, as shown by the in Table 1b, the two-stage model does not capture long-range structure as well as a MelNet model that is trained without access to any intermediate representations. To isolate the impact of multiscale modelling procedure described in Section 5, we train models with varying numbers of tiers and evaluate the long-term coherence of their respective samples. As noted before, long-term coherence is difficult to quantify and we provide samples on the accompanying web page so that the reader can make their own judgements. We believe the samples clearly demonstrate that increasing the number of tiers in samples with more coherent high-level structure. We note that our experiment varies the number of tiers from two to five. Training a single-tier model on full-resolution spectrograms was prohibitively expensive in terms of memory consumption. This highlights another benefit of multiscale modelling-large, deep networks can be allocated to learning complex distributional structure in the initial tiers while shallower networks can be used for modelling the relatively simple, low-entropy distributions in the upsampling tiers. This allows multiscale models to effectively allocate network capacity in proportion to the complexity of the modelling task. The predominant line of research regarding generative models for audio has been directed towards modelling time-domain waveforms with autoregressive models (van den a; ; . WaveNet is a competitive baseline for audio generation, and as such, is used for comparison in many of our experiments. However, we note that the contribution of our work is in many ways complementary to that of WaveNet. MelNet is more proficient at capturing high-level structure, whereas WaveNet is capable of producing higher-fidelity audio. Several works have demonstrated that time-domain models can be used to invert spectral representations to highfidelity audio (; ; Arık et al., 2019), suggesting that MelNet could be used in concert with time-domain models such as WaveNet. and capture long-range dependencies in waveforms by utilizing a hierarchy of autoencoders. This approach requires multiple stages of models which must be trained sequentially, whereas the multiscale approach in this work can be parallelized over tiers. Additionally, these approaches do not directly optimize the data likelihood, nor do they admit tractable marginalization over the latent codes. We also note that the modelling techniques devised in these works can be broadly applied to autoregressive models such as ours, making their contributions largely complementary to ours. Recent works have used generative adversarial networks (GANs) to model both waveforms and spectral representations. As with image generation, it remains unclear whether GANs capture all modes of the data distribution. Furthermore, these approaches are restricted to generating fixed-duration segments of audio, which precludes their usage in many audio generation tasks. Generating spectral representations is common practice for end-to-end text-to-speech models (; ;). However, these models use probabilistic models which are much less expressive than the fine-grained autoregressive model used by MelNet. Consequently, these models are unsuitable for modelling high-entropy, multimodal distributions such as those involved in tasks like unconditional music generation. The network architecture used for MelNet is heavily influenced by recent advancements in deep autoregressive models for images. introduced an LSTM architecture for autoregressive modelling of 2D images and van den Oord et al. (2016c) introduced PixelRNN and PixelCNN and scaled up the models to handle the modelling of natural images. Subsequent works in autoregressive image modelling have steadily improved state-of-the-art for image density estimation (van den b; ; ;). We draw inspiration from many of these models, and ultimately design a recurrent architecture of our own which is suitable for modelling spectrograms rather than images. We note that our choice of architecture is not a fundamental contribution of this work. While we have designed the architecture particularly for modelling spectrograms, we did not experimentally validate whether it outperforms existing architectures and make no such claims to this effect. We use a multidimensional recurrence in both the time-delayed stack and the upsampling tiers to extract features from two-dimensional inputs. Our multidimensional recurrence is effectively'factorized' as it independently applies one-dimensional RNNs across each dimension. This approach differs from the tightly coupled multidimensional recurrences used by MDRNNs and GridLSTMs and more closely resembles the approach taken by ReNet . Our approach allows for efficient training as we can extract features from an M × N grid in max(M, N) sequential recurrent steps rather than the M + N sequential steps required for tightly coupled recurrences. Additionally, our approach enables the use of highly optimized one-dimensional RNN implementations. Various approaches to image generation have succeeded in generating high-resolution, globally coherent images with hundreds of thousands of dimensions (; ;). The methods introduced in these works are not directly transferable to waveform generation, as they exploit spatial properties of images which are absent in one-dimensional audio signals. However, these methods are more straightforwardly applicable to two-dimensional representations such as spectrograms. Of particular relevance to our work are approaches which combine autoregressive models with multiscale modelling (van den c; ; ;). Our work demonstrates that the benefits of a multiscale autoregressive model extend beyond the task of image generation, and can be used to generate high-resolution, globally coherent spectrograms. We have introduced MelNet, a generative model for spectral representations of audio. MelNet combines a highly expressive autoregressive model with a multiscale modelling scheme to generate high-resolution spectrograms with realistic structure on both local and global scales. In comparison to previous works which model time-domain signals directly, MelNet is particularly well-suited to model long-range temporal dependencies. Experiments show promising across a diverse set of audio generation tasks. Furthermore, we believe MelNet provides a foundation for various directions of future work. Two particularly promising directions are text-to-speech synthesis and representation learning: • Text-to-Speech Synthesis: MelNet utilizes a more flexible probabilistic model than existing end-to-end text-to-speech models, making it well-suited to model expressive, multi-modal speech data. • Representation Learning: MelNet is able to uncover salient structure from large quantities of unlabelled audio. Large-scale, pre-trained autoregressive models for language modelling have demonstrated significant benefits when fine-tuned for downstream tasks. Likewise, representations learned by MelNet could potentially aid downstream tasks such as speech recognition.

represent the short summary for finding the corresponding scientific article (long)

We introduce an autoregressive generative model for spectrograms and demonstrate applications to speech and music generation

We investigate a variant of variational autoencoders where there is a superstructure of discrete latent variables on top of the latent features.


Question: Represent the scientific article to find a short TLDR

Question: Deep convolutional network architectures are often assumed to guarantee generalization for small image translations and deformations. In this paper we show that modern CNNs (VGG16, ResNet50, and InceptionResNetV2) can drastically change their output when an image is translated in the image plane by a few pixels, and that this failure of generalization also happens with other realistic small image transformations. Furthermore, we see these failures to generalize more frequently in more modern networks. We show that these failures are related to the fact that the architecture of modern CNNs ignores the classical sampling theorem so that generalization is not guaranteed. We also show that biases in the statistics of commonly used image datasets makes it unlikely that CNNs will learn to be invariant to these transformations. Taken together our suggest that the performance of CNNs in object recognition falls far short of the generalization capabilities of humans. Deep convolutional neural networks (CNNs) have revolutionized computer vision. Perhaps the most dramatic success is in the area of object recognition, where performance is now described as "superhuman" . A key to the success of any machine learning method is the inductive bias of the method, and clearly the choice of architecture in a neural network significantly affects the inductive bias. In particular, the choice of convolution and pooling in CNNs is motivated by the desire to endow the networks with invariance to irrelevant cues such as image translations, scalings, and other small deformations (; BID33 . This motivation was made explicit in the 1980s by Fukushima in describing the "neocognitron" architecture, which served as inspiration for modern CNNs , "After finishing the process of learning, pattern recognition is performed on the basis of similarity in shape between patterns, and is not affected by deformation, nor by changes in size, nor by shifts in the position of the input patterns." Despite the excellent performance of CNNs on object recognition, the vulnerability to adversarial attacks suggests that superficial changes can in highly non-human shifts in prediction (e.g. BID1 BID27 . In addition, filtering the image in the Fourier domain (in a way that does not change human prediction) also in a substantial drop in prediction accuracy BID13. These and other BID20 indicate that CNNs are not invariant to cues that are irrelevant to the object identity. An argument against adversarial attacks on CNNs is that they often involve highly unnatural transformations to the input images, hence in some sense we would not expect CNNs to be invariant to these transformations. When considering more natural transformations, there is preliminary evidence that AlexNet BID15 ) is robust to some of them BID33. On the other hand, there is also preliminary evidence for lack of robustness in the more modern networks for object classification BID2 and detection BID21 along with studies suggesting that with small CNNs and the MNIST data, data augmentation is the main feature affecting CNN invariance BID14 ). An indirect method to probe for invariances measures the linearity of the learned representations under natural transformations to the input image (Lenc Figure 1 : Examples of jagged predictions of modern deep convolutional neural networks. Top: A negligible vertical shift of the object (Kuvasz) in an abrupt decrease in the network's predicted score of the correct class. Middle: A tiny increase in the size of the object (Lotion) produces a dramatic decrease in the network's predicted score of the correct class. Bottom: A very small change in the bear's posture in an abrupt decrease in the network's predicted score of the correct class. Colored dots represent images chosen from interesting x-axis locations of the graphs on the right. These dots illustrate sensitivity of modern neural networks to small, insignificant (to a human), and realistic variations in the image. BID17 BID12; BID6. The recent work of BID10 investigates adversarial attacks that use only rotations and translations. They find that "simple transformations, namely translations and rotations alone, are sufficient to fool neural network-based vision models on a significant fraction of inputs" and show that advanced data augmentation methods can make the networks more robust. In this paper, we directly ask "why are modern CNNs not invariant to natural image transformations despite the architecture being explicitly designed to provide such invariances?". Specifically, we systematically examine the invariances of three modern deep CNNs: VGG-16 BID26, ResNet-50 , and InceptionResNet-V2 BID28. We find that modern deep CNNs are not invariant to translations, scalings and other realistic image transformations, and this lack of invariance is related to the subsampling operation and the biases contained in image datasets. Figure 1 contains examples of abrupt failures following tiny realistic transformations for the InceptionResNet-V2 CNN. Shifting or scaling the object by just one pixel could in a sharp change in prediction. In the top row, we embed the original image in a larger image and shift it in the image plane (while filling in the rest of the image with a simple inpainting procedure). In the middle row, we repeat this protocol with rescaling. In the bottom row, we show frames from a BBC film in which the ice bear moves almost imperceptibly between frames and the network's output changes dramatically 1. In order to measure how typical these failures are, we randomly chose images from the ImageNet validation set and measured the output of three modern CNNs as we embedded these images in a larger image and systematically varied the vertical translation. As was the case in figure 1, we used a simple inpainting procedure to fill in the rest of the image. Results are shown in figure 2. Each row corresponds to an image under different translations and the color denotes the network's estimate of the probability of the correct class. Thus a row that is all light corresponds to a correct classification that is invariant to translation, while a row that is all dark corresponds to an incorrect classification that is invariant to translation. Surprisingly, many rows show abrupt transitions from light to dark, indicating that the classification changes abruptly as the object is translated. We quantify the lack of invariance by a measure we call "jaggedness": the number of times the network's predictions had the correct class in its top-5 and after just one pixel shift it moved outside of the top-5 (and also the opposite transition from non-top-5 to top5). Using this measure, we find that for approximately 30% of the images, the output is "jagged", i.e the network changes its prediction by a shift of a single pixel. Also, as shown in the right of figure 2, jaggedness is greater for the modern, deeper, networks compared to the less modern VGG16 network. While the deeper networks have better test accuracy, they are also less invariant. In the appendix we also show an alternative to the "jaggedness" measure, which gives similar . A natural criticism of these is that they are somehow related to the image resizing and inpainting procedures that we used. To test this possibility, we repeated the experiment with a different protocol where we chose different crops of the original ImageNet image while making sure that the object bounding box remained within the crop. This protocol does not require any inpainting while still translating the object location within the new image. Results are shown in the appendix. We still have a large fraction of images for which the prediction is not invariant to translation. We also show in the appendix similar for scaling rather than translation. Overall we find that regardless of the protocol used, modern CNNs often change their output significantly as a of a small translation or scaling. The failure of CNNs to generalize to image translations is particularly puzzling. Intuitively, it would seem that if all layers in a network are convolutional then the representation should simply translate when an image is translated. If the final features for classification are obtained by a global pooling operation on the representation (as is done for example in ResNet50 and InceptionResNetV2) then these features should be invariant to translation. Where does this intuition fail? This intuition ignores the subsampling operation which is prevalent in modern CNNs, also known as "stride". This failure of translation invariance in systems with subsampling was explicitly discussed in Simoncelli et al. BID25 who wrote "We cannot literally expect translation invariance in a system based on convolution and subsampling: translation of the input signal cannot produce simple translations of the transform coefficients, unless the translation is a multiple of each of the subsampling factors in the system". Since deep networks often contain many subsampling operations, the subsampling factor of the deep layers may be very large so that "literal" translation invariance only holds for very special translations. In InceptionResnetV2, for example, the subsampling factor is 60, so we expect exact translation invariance to hold only for 1 60 2 of possible translations. Simoncelli et al. also defined a weaker form of translation invariance, which they called "shiftability" and showed that it can hold for systems with subsampling (this is related to weak translation invariance as defined by BID17, see also BID11 BID6 for related ideas applied to neural networks). Here we extend the basic shiftability to show that when shiftability holds, then global pooling will indeed yield invariant representations. We define r(x) as the response of a feature detector at location x in the image plane. We say that this response is "convolutional" if translating the image by any translation δ yields a translation of the response by the same δ. This definition includes cases when the feature response is obtained by convolving the input image with a fixed filter, but also includes combinations of linear operations and nonlinear operations that do not include any subsampling. We start by a trivial observation:Observation: If r(x) is convolutional then global pooling r = x r(x) is translation invariant. Proof: This follows directly from the definition of a convolutional response. If r(x) is the feature response to one image and r 2 (x) is the feature response to the same image translated, then x r(x) = x r 2 (x) since the two responses are shifts of each other. Definition: A feature detector r(x) with subsampling factor s is called "shiftable" if for any x the detector output at location x can be linearly interpolated from the responses on the sampling grid: DISPLAYFORM0 where x i are located on the sampling grid for subsampling factor s and B s (x) is the basis function for reconstructing r(x) from the samples. The classic Shannon-Nyquist theorem tells us that r(x) will be shiftable if and only if the sampling frequency is at least twice the highest frequency in r(x).Claim: If r(x) is shiftable then global pooling on the sampling grid r = i r(x i) is translation invariant. Proof: This follows from the fact that global pooling on the sampling grid is (up to a constant) the same as global pooling for all x. DISPLAYFORM1 DISPLAYFORM2 where DISPLAYFORM3 While the claim focuses on a global translation, it can also be extended to piecewise constant transformations. Corollary: Consider a set of transformations T that are constant on a set of given image subareas. If r(x) is shiftable and for a given image, the support of r(x) and its receptive field is contained in the same subregion for all transformations in T, then global pooling on the sampling grid is invariant to any transformation in T.Proof: This follows from the fact that applying any transformation in T to an image has the same effect on the feature map r(x) as translating the image. To illustrate the importance of the sampling theorem in guaranteeing invariance in CNNs, consider a convolutional layer in a deep CNN where each unit acts as a localized "part detector" (this has been reported to be the case for many modern CNNs BID33 BID34). Each such part detector has a spatial tuning function and the degree of sharpness of this tuning function will determine whether the feature map can be subsampled while preserving shiftability or not. For example, consider a part detector that fires only when the part is exactly at the center of its receptive field. If there is no subsampling, then as we translate the input image, the feature map will translate as well, and the global sum of the feature map is invariant to translation. But if we subsample by two (or equivalently use a stride of two), then there will only be activity in the feature map when the feature is centered on an even pixel, but not when it is centered on an odd pixel. This means that the global sum of the feature map will not be invariant to translation. In the language of Fourier transforms, the problem with a part detector that fires only when the part is exactly at the center of the receptive field is that the feature map contains many high frequencies and hence it cannot be subsampled while preserving shiftability. On the other hand, if we have a part detector whose spatial tuning function is more broad, it can be shiftable and our claim (above) shows that the global sum of activities in a feature map will be preserved for all translations, even though the individual firing rates of units will still be different when the part is centered at an odd pixel or an even pixel. Our corollary (above), shows the importance of shiftability to other smooth transformations: in this case each "part detector" will translate with a different translation but it is still the case that nonshiftable representations will not preserve the global sum of activities as the image is transformed, while shiftable representations will. FIG1 examines the extent to which the representations learned by modern CNNs are invariant or shiftable. The top row shows an image that is translated vertically, while the bottom three rows show the representations in different layers for the three CNNs we consider. For VGG16 the representation appears to shift along with the object, including the final layer where the blurred pattern of response is not a simple translation of the original response, but seems to preserve the global sum for this particular image. For the two more modern networks, the responses are sharper but lose their shiftability in the later layers. In particular, the final layers show approximate invariance to one special translation but no response at all to another translation, suggesting that the many layers of subsampling yield a final response that is not shiftable. We also performed a more quantitative measure of shiftability by counting for a given image the number of times the global sum of activities in each layer changes significantly (more than 20% of mean) as the input is shifted (for each image, we only considered feature maps where the maximum response was above a threshold). We call this measure "nonshiftability". According to the preceding analysis, in architectures that obey the sampling theorem, the global sum should be invariant to input translation so nonshiftability should be zero in all layers. We find that for all three networks, the initial layers have nonshiftability close to zero but as we go deeper and deeper nonshiftability increases. Furthermore, the deeper, more modern networks, exhibit larger nonshiftability in their deep layers compared to VGG16 (see appendix for graphs).How can we guarantee that representations in CNNs will be shiftable? As explained above, we need to make sure that any feature map that uses stride does not contain frequencies above the Nyquist frequency. If CNNs were purely linear, we could simply blur the input images so that they would not include any frequencies higher than the Nyquist limit determined by the final sampling factor of the network. But since CNNs also include nonlinearities, they can add high frequencies that were not present in the input. An important message of the sampling theorem is that you should always blur before subsampling. Translated to the language of neural networks this means that stride (i.e. subsampling) should always be combined with pooling (i.e. blurring) in the preceding layer. Indeed if we have an arbitrarily deep CNN where all the layers use stride=1 followed by one layer that has a stride greater than one, then by choosing the pooling window appropriately we can guarantee that the final layer will still be shiftable. If we do not use appropriate pooling then there is no guarantee that this layer will be shiftable. Even if we use appropriate pooling that ensures that a given layer is shiftable, the subsequent nonlinearities in a CNN may not preserve the shiftability, as the nonlinearities may again introduce high frequencies. To illustrate the effect of pooling on shiftability in modern CNNs we replaced the 2 × 2 max pooling layers of VGG16 with 6 × 6 average pooling. This has the effect of reducing low freqencies but given the nonlinearities, it does not guarantee shiftability. As shown in figure 4 this simple change makes the representations approximately shiftable and, as predicted by our theory, the global sum is now invariant to both translations and rescalings of the input. This invariance of course comes with a price: the feature maps now have less detail and in preliminary experiments we find that recognition performance decreases somewhat. But the sampling theorem tells us that if we want to use subsampling while avoiding aliasing, we need to ensure that no high frequencies (relative to the Nyquist frequency) are present in the feature maps. As an alternative to pooling, Ruderman et al. BID22 have shown that networks may learn smooth filters that will lead to reduced sensitivity to transformations. Evidently, the filters learned in standard VGG16 are not smooth enough. While the preceding discussion suggests that the CNN architecture will not yield translation invariance "for free", there is still the possibility that the CNN will learn a translation invariant prediction from the training examples. This requires that the training set will actually be invariant to the irrelevant transformations. We examined the degree of invariance in the ImageNet training set by manually labeling the training images in five categories: Tibetan terrier, elephant, pineapple, bagel and rowing paddle. Consistent with previous on "dataset bias" BID24 BID19 BID0 BID29 BID30 BID18 we 2 × 2 max 6 × 6 average Translation: find that the ImageNet dataset is not invariant to translations and rescalings. FIG3 shows the distribution of the distances between the eyes of a "Tibetan terrier" and the positions of the center point between the dog's eyes. Notice that both distributions are far from uniform. Similar are obtained for the other four categories. To be more quantitative, we used the available bounding-box labels, and extracted the center point of the bounding-box and its height as proxies for the object position and size respectively. We then applied a statistical significance test to ask whether object location and object sizes were uniform for that category. For more than 900 out of the 1000 categories we found that location and size were highly non uniform (P < 10 −10). Given these strong biases, we cannot expect a learning system to learn to be invariant. DISPLAYFORM0 Even if the training set is not invariant, we can make it invariant using data augmentation. Will this make the CNN learn an invariant prediction? First, we note that we used pretrained networks and according to the authors' description of the training procedure, all three networks were trained using data augmentation. Obviously, not any data augmentation is sufficient for the networks to learn invariances. To understand the failure of data augmentation, it is again instructive to consider the subsampling factor. Since in modern networks the subsampling factor is approximately 60, then for a system to learn complete invariance to translation only, it would need to see 60 2 = 3600 augmented versions of each training example, or it would need to have an inductive bias that allows it to generalize over transformations. If we also add invariance to rotations and scalings, the number grows exponentially with the number of irrelevant transformations. Engstrom et al. BID10 suggest a sophisticated data augmentation method and show that it increases the invariance to translation and rotation. However, for challenging datasets such as ImageNet the lack of invariance largely persists. Figure 6: The performance of modern CNNs on test images from ImageNet that are embedded in a random location in a larger image is quite poor (less than 50% accuracy). Human performance is not affected. Right: An example of a full sized image and the same image resized to 100x100. Although our show that modern CNNs fail to generalize for small image transformations, their performance on the ImageNet test set is still amazingly good and far better than previous techniques. This is related to the fact that the ImageNet test set contains the same photographer's biases as the training set, so generalization to very different sizes and locations is not required. To highlight this point, we created a new test set in which ImageNet images were embedded in a larger image in a random location (and the missing pixels were filled in using a simple inpainting algorithm). Figure 6 shows that human performance is not affected by the rescaling and random translations, while the performance of modern CNNs deteriorates dramatically. In fact, when images are scaled to half their original size and randomly translated, the accuracy of modern CNNs is less than 50%, typically considered poor performance. One way in which modern systems partially address the lack of invariance is using test time augmentation in which the system output on a given image is computed by a majority vote among many random crops of the image. Clearly this is wasteful in resources and still only provides partial invariance.6 DISCUSSION CNN architectures were designed based on an intuition that the convolutional structure and pooling operations will give invariance to translations and small image deformations "for free". In this paper we have shown that this intuition breaks down once subsampling, or "stride" is used and we have presented empirical evidence that modern CNNs do not display the desired invariances since the architecture ignores the classic sampling theorem. This still leaves open the possibility of a CNN learning invariance from the data but we have shown that the ImageNet training and testing examples include significant photographer's bias so that it is unlikely that a system will learn invariance using these examples. In addition to pointing out these failures, the sampling theorem also suggests a way to impose translation invariance by ensuring that all representations are sufficiently blurred to overcome the subsampling. However, such blurred representations may lead to a decrease in performance, especially in datasets and benchmarks that contain photographer's bias. Alternatively, one could use specially designed features in which invariance is hard coded or neural network architectures that explicitly enforce invariance BID23; BID3 BID20 BID9 BID32 BID31 BID5. Again, as long as the datasets contain significant photographer's bias, such invariant approaches may lead to a decrease in performance. We download this video: https://www.youtube.com/watch?v=0mgnf6t9VEc using an online downloader. We load the video frames and resize them to 299 by 299 as used by the standard Keras applications framework (https://keras.io/applications/). We preprocess the frames using the standard Keras preprocessing function. Finally, we use the predictions of the InceptionV3 model to demonstrate the jagged behavior shown in figure 1. Top-1 Top-5 Parameters Depth VGG16 BID26 0 9** 5HV1HW,QFHSWLRQ5HV1HW9 0$' Figure 7: We also measure jaggedness using the Mean Absolute Difference (MAD) in the probability of the correct response as the image is shifted by a single pixel. Results are similar to those using the jaggedness measure described in the text. Figure 10: Nonshiftability as a function of depth in the three networks. Nonshiftability is defined as the number of times the global sum of a feature map changes by more than 20% of the mean as the input is translated. We only consider feature maps where the maximum response was above a threshold. According to our analysis, this measure should be zero if the representation is shiftable. Each line shows the nonshiftability in different layers in response to a randomly selected image.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Modern deep CNNs are not invariant to translations, scalings and other realistic image transformations, and this lack of invariance is related to the subsampling operation and the biases contained in image datasets.

We introduce a statistical approach to assessing neural network robustness that provides an informative notion of how robust a network is, rather than just the conventional binary assertion of whether or not of property is violated.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: We present a real-time method for synthesizing highly complex human motions using a novel training regime we call the auto-conditioned Recurrent Neural Network (acRNN). Recently, researchers have attempted to synthesize new motion by using autoregressive techniques, but existing methods tend to freeze or diverge after a couple of seconds due to an accumulation of errors that are fed back into the network. Furthermore, such methods have only been shown to be reliable for relatively simple human motions, such as walking or running. In contrast, our approach can synthesize arbitrary motions with highly complex styles, including dances or martial arts in addition to locomotion. The acRNN is able to accomplish this by explicitly accommodating for autoregressive noise accumulation during training. Our work is the first to our knowledge that demonstrates the ability to generate over 18,000 continuous frames (300 seconds) of new complex human motion w.r.t. different styles. The synthesis of realistic human motion has recently seen increased interest BID12 BID39 BID4 BID14 BID1 BID25 with applications beyond animation and video games. The simulation of human looking virtual agents is likely to become mainstream with the dramatic advancement of Artificial Intelligence and the democratization of Virtual Reality. A challenge for human motion synthesis is to automatically generate new variations of motions while preserving a certain style, e.g., generating large numbers of different Bollywood dances for hundreds of characters in an animated scene of an Indian party. Aided by the availability of large human-motion capture databases, many database-driven frameworks have been employed to this end, including motion graphs BID18 BID33 BID27, as well as linear BID34 BID2 BID36 and kernel methods BID29 BID31 BID8 BID28 BID42, which blend key-frame motions from a database. It is hard for these methods, however, to add new variations to existing motions in the database while keeping the style consistent. This is especially true for motions with a complex style such as dancing and martial arts. More recently, with the rapid development in deep learning, people have started to use neural networks to accomplish this task BID13. These works have shown promising , demonstrating the ability of using high-level parameters (such as a walking-path) to synthesize locomotion tasks such as jumping, running, walking, balancing, etc. These networks do not generate new variations of complex motion, however, being instead limited to specific use cases. In contrast, our paper provides a robust framework that can synthesize highly complex human motion variations of arbitrary styles, such as dancing and martial arts, without querying a database. We achieve this by using a novel deep auto-conditioned RNN (acRNN) network architecture. Recurrent neural networks are autoregressive deep learning frameworks which seek to predict sequences of data similar to a training distribution. Such a framework is intuitive to apply to human motion, which can be naturally modeled as a time series of skeletal joint positions. We are not the first to leverage RNNs for this task BID4 BID14 BID1 BID25, and these works produce reasonably realistic output at a number of tasks such as sitting, talking, smoking, etc. However, these existing methods also have a critical drawback: the motion becomes unrealistic within a couple of seconds and is unable to recover. This issue is commonly attributed to error accumulation due to feeding network output back into itself BID13. This is reasonable, as the network during training is given ground-truth input sequences to condition its subsequent guess, but at run time, must condition this guess on its own output. As the output distribution of the network will not be identical to that of the ground-truth, it is in effect encountering a new situation at test-time. The acRNN structure compensates for this by linking the network's own predicted output into its future input streams during training, a similar approach to the technique proposed in BID0. Our method is light-weight and can be used in conjunction with any other RNN based learning scheme. Though straightforward, this technique fixes the issue of error accumulation, and allows the network to output incredibly long sequences without failure, on the order of hundreds of seconds (see Figure 5). Though we are yet as unable to prove the permanent stability of this structure, it seems empirically that motion can be generated without end. In summary, we present a new RNN training method capable for the first time of synthesizing potentially indefinitely long sequences of realistic and complex human motions with respect to different styles. Many approaches have developed over the years in order to generate realistic human motion. In this section, we first review the literature that has dealt with motion synthesis using simulation methods and database-driven methods, then review the more recent deep learning approaches. Simulation-based Methods. Simulation-based techniques are able to produce physically plausible animations BID5 BID3 BID10 BID9 BID24, including realistic balancing, motion on terrain of various heights, and recovery from falling. Many of these methods consider physical constraints on the human skeleton while optimizing an motion objective. For example, in the work of, one task they employ is moving the skeleton in a certain direction without falling over. Similarly, in Ha et al. BID9, given initial fall conditions, they seek to minimize joint stress due to landing impact while ensuring a desired landing pose. Though the output is plausible, they require specific objectives and constraints for each individual task; it is infeasible to apply such explicit priors to highly stylized and varied motion such as dancing or martial arts. There are also some recent works BID32 BID26 which attempt to use less rigid objectives, instead employing adversarial and reinforcement learning. However, these motions often look uncanny and not human-like. Database-driven Methods. Motion graphs BID18 BID33 BID27, which stitch transitions into segments queried from a database, can generate locomotion along arbitrary paths, but are in essence limited to producing memorized sequences. Frameworks based on linear BID34 BID2 BID36 and kernal methods BID29 BID31 BID8 BID28 BID42 have also shown reasonable success. Taylor et al. BID37 use conditional restricted boltzmann machines to model motion styles. Grochow et al. BID8 and BID41 both use Gaussian Process Latent Variable Models (GPLVM) to control motion. apply reinforcement learning in a reduced space to compute optimal motions for a variety of motion tasks, including movement, punching, and kicking. Kee et al. BID20 use a representation of motion data called motion-fields which allows users to interactively control movement of a character. The work of Xia et al. BID44 is able to achieve real-time style transfers of unlabeled motion. Taylor et al. BID38 use binary latent variables which are connected between different time steps while Liu et al. BID23 estimate physical parameters from motion capture data. Database methods have limitations in synthesizing new variations of motion, in addition to the high memory costs for storing complex motions. Deep Learning Approaches. The use of recurrent networks is a natural approach to dealing with the problem of human motion. RNNs are trained to generate output sequentially, each output conditioned on the previous elements in the sequence. These networks have shown much success in Natural Language Processing for generating text BID35, hand written characters BID6 BID7, and even captioning images BID40. For the purpose of motion prediction and generation, Fragkiadaki et al. BID4 propose to use encoderrecurrent-decoder (ERD), jointly learning a skeleton embedding along with sequential information, in order to make body positions more realistic. In Jain et al. BID14, the authors employ RNNs to learn spatio-temporal graphs of interaction, a structure which naturally applies to the human skeleton over time-joint positions are related over consecutive frames, and joints also interact with each other spatially (arms and legs interact with each other, the spine interacts with all other joints). Bütepage et al. BID1 attempt to learn dance sequentially, but they are unable to produce varied and realistic output. More recently, Martinez et al. BID25 propose using a sequence-to-sequence architecture along with sampling-based loss. A main problem with these approaches, even in the case of the ERD, is that motion generation tends to converge to a mean position over time. Using our proposed training method, motion neither halts nor becomes unrecognizable for any period of time. Holden et al. BID11 demonstrate that a manifold of human motion can be learned using an autoencoder trained on the CMU dataset. They BID12 extend this work by using this deep convolutional auto-encoder, in addition to a disambiguation network, in order to produce realistic walking motion along a user-defined path. They are also able to use the embedding space of the encoder to perform style transfer across different walks. Using the same autoencoder and a different disambiguation network, they can also perform other user defined tasks such as punching and kicking. Their method does not use an RNN structure and so it can only generate fixed length sequences based on the architecture of the disambiguation network. More recently, Holden et al. BID13 ) use a Phase-Functioned Neural Network which takes the geometry of the scene into account to produce motion along a user-defined path. This method makes use of humans' periodic change of gait to synthesize realistic walking motion but has not demonstrated the ability to generate motions that have more complex step movements. Auto-Conditioned RNN. Recurrent neural networks are well documented in the literature. A good survey and introduction can be found at BID16 BID30. In short, a recurrent network can be represented as a function with a hidden memory unit, x t+1 = f (x t, m t), where m t is the "memory" of the network, and is updated during every forward pass, and initialized at 0. The motivation is that the memory stores important information about a sequence up until that point, which can help with the prediction of the next element. In the experiments that follow, we use a special type of RNN called an "LSTM", or a "long short term memory" network. We refer to the network trained with our method as "acLSTM". As mentioned in the introduction, the major drawback of using LSTM/RNN deep learning methods for motion prediction is the problem of error accumulation. Following the conventional way to train an RNN, the network is recursively given a sequence of ground truth motion data, DISPLAYFORM0 and asked to produce the output G 2,k+1 [g 2, ..., g k+1]. Specifically, at training time, the recursive module at time step t, M t, is conditioned on the input [g 1, ..., g t−1], and the error is measured between its output and g t. Because of this, the backpropogation algorithm BID43 used for updating the parameters will always optimize w.r.t. the input ground-truth sequences [g 1, ..., g t−1]. The parameters are accustomed to ground truth input -something it does not have access to at test-time. It is easy to see why problems will emerge: even if initial input is similar to ground truth, those slight differences will accumulate as the output is fed back in, producing output that become progressively worse until the sequence diverges or freezes. Effectively, the network is encountering a completely novel situation during test time as compared to training time, and so cannot perform well. The issue is so prevalent, in fact, that previous methods fail to produce realistic output after just several seconds BID4 BID14 BID25.Holden et al. BID11 show that an autoencoder framework can to some degree "fix" such broken input, and some researchers have tried jointly learning such an encoder-decoder network alongside RNNs to better condition subsequent input BID7 BID4. However, in the case of this framework being applied to motion as in ERD BID4, it does not generalize to indefinitely long sequences, as shown in BID14. It seems as though the autoencoder might mitigate error accumulation, but does not eliminate it. The acRNN, on the other hand, deals with poor network output explicitly by using it during training. Instead of only feeding in ground-truth instances, we use subsequences of the network's own outputs at periodic intervals. For instance, sticking with the example above, instead of conditioning the network on DISPLAYFORM1 The variable p u+1 is the network output conditioned on [g 1, ..., g u], and p u+2 is conditioned on [g 1, ..., g u, p u+1]. In this example, we refer to v as the "condition length" and u as the "ground-truth length". As the network is conditioned on its own output during training, it is able to deal with such input during synthesis. FIG0 details an unrolled Auto-Conditioned RNN with condition length u = v = 4, and FIG0 shows a more detailed view or our network. The method of BID0 also proposes using network output during training, but does so stochastically, without fixing condition lengths. However, we found that changing the condition/ground-truth length while keeping the proportion of ground-truth input fixed affects both the accuracy and variation of the output. See Figure 9 in the appendix. Auto-conditioning also has the interpretation of training the network to produce longer sequences without further input. Whereas with standard training the network error is measured only against p u+1 when conditioned with [g 1, ..., g u], under auto-conditioning the error is computed on the entire sequence p u+1,..., p u+v w.r.t. the same input. This effectively forces the network to produce v frames of output simultaneously as opposed to only one. Martinez et al. BID25 ) also use contiguous sequences of network output during training, but unlike us they do not alternate these with ground-truth input at regular intervals. Data Representation. We use the publicly available CMU motion-capture dataset for our experiments. The dataset is given as sequences of 57 skeleton joint positions in 3D-space. First, we define a root joint, whose position at time t is given by r t = (r 1,t, r 2,t, r 3,t). In order to better capture relative motion, we instead use the displacement of the root from the previous frame for input -r t = (r 1,t − r 1,t−1, r 2,t − r 2,t−1, r 3,t − r 3,t−1). For every other joint at time t, with position j t = (j 1,t, j 2,t, j 3,t), we represent it as as the relative distance in the world-coordinate system to the root joint,j t = (j 1,t − r 1,t, j 2,t − r 2,t, j 3,t − r 3,t). All distances are stored in meters. We use a skeleton with height 1.54 meters in neutral pose for all experiments. We found this representation to be desirable for several reasons. Primarily, if there is periodic motion in the dataset, we would like frames at the same point in the repeated activity to have small Euclidean distance. If we instead used absolute positions, even if it were only for the hip, this would certainly not be the case. We note that there are alternative representations which achieve the same property. BID4 express joint positions as rotations relative to a parent joint, and BID12 define them in the body's relative coordinate system along with a relative rotation of the body w.r.t. the previous frame. Training. We train the acLSTM with three fully connected layers with a memory size of 1024, similar to BID4 BID14 BID1. The main difference is that for every u ground-truth inputs of the time series, we connect v instances of the network's own output into its subsequent input streams (see section 2.1). In the main body of the paper, we set u = v = 5. We carry out further experiments with varying u and v in the appendix. We train with a sequence length of 100 for 500000 iterations using the ADAM backpropogation algorithm BID17 on an NVIDIA 1080 GPU for each dataset we experiment on. We use Euclidean loss for our objective function. The initial learning rate is is set to 0.0001. We implement the training using the python caffe framework BID15. We sample sequences at multiple frame-rates as well as rotate the sequence randomly in order to increase the training size. In detail, if at time t we input the network H with ground truth, then the loss is given by: DISPLAYFORM2 where x t and x t+1 are the ground truth motions for time steps t and t + 1.If at time t the network is input with its own previous output, the loss is given by: DISPLAYFORM3 DISPLAYFORM4. k indicates how many times the network has fed itself is its own input since the last injection of ground-truth. It is bounded by the condition length (see previous section). We evaluate our synthesized motion on different networks trained on each of four distinct subsets from the CMU motion capture database: martial arts, Indian dance, Indian/salsa hybrid, and walking. An anonymous video of the can be found here: https://youtu.be/FunMxjmDIQM.Quantitative Results. TAB0 shows the prediction error as Euclidean distance from the ground truth for different motion styles at various time frames. We compare with a 3-layer LSTM (LSTM-3LR), ERD BID4, the seq2seq framework of BID25 as well as scheduled sampling BID0. Though our goal is long-term stable synthesis and not prediction, it can be seen in TAB0 that acLSTM performs reasonably well in this regard, even achieving the best performance at most time-scales for the Indian dance category. The method of BID25 performs the best in the short term, but has the worst error by far past the half-second mark. As noted in BID4, the stochasticity of human motion makes longer-term prediction infeasible. TAB3 in the appendix shows this error difference between versions of the Indian dance network trained with different condition lengths. Figure 2 shows the average change between subsequent frames for different frame times for the acLSTM and the basic scheme. For the more complex motions of Martial Arts and Indian Dance, it is clear from the graph that acLSTM continues producing motion in the long-term while the basic training scheme (LSTM-3LR) without auto-conditioning in stagnated motion, as the network freezes into a converged mean position. Likewise, Figure 9 in the appendix shows this average change for the Indian dance network trained with different condition lengths. We note that while the methods of BID4 BID25 do not simply freeze completely, as with the basic scheme, their motion becomes unrealistic at around the same time FIG3. This is consistent with the observations of the original authors. Qualitative Results. Figure 4 shows several example frames taken from 50 second synthesized outputs, representing both the extended long term complexity and plausibility of the output. In comparison, our implementations of ERD, and seq2seq are only able to generate motion for a couple:DONLQJ /670/5 DF/670/5ZFOHQJWK,QGLDQ'DQFH )UDPH1XPEHU 0DUWLDO$UWV of seconds before they become unrealistic FIG3 ). Scheduled sampling BID0 performs better than ERD and seq2seq, but also freezes eventually, perhaps because it does not force the network to generate consistently longer sequences during training as our method does. We also demonstrate the possibility of creating hybrid motions by mixing training sets in third row of Figure 4.It should be noted that the motion in our framework, while never permanently failing, also does not remain perfectly realistic, and short-term freezing does sometimes occur. This, however, does not occur for the network trained just on walking. It is perhaps that the movement of the feet in conjunction with the absolute movement of the root joint is not as easy to discern when the feet leave the ground aperiodically, or there are jumping motions. BID4 0 BID4 0.32 0.44 0.63 0.90 1.14 1.40 1.61 1.88 seq2seq BID25 0 Figure 4: Motion sequences generated by acLSTM, sampled at various frames. Motion style from top to bottom: martial arts, Indian dancing, Indian/salsa hybrid and walking. All the motions are generated at 60 fps, and are initialized with 10 frames of ground truth data randomly picked up from the database. The number at the bottom of each image is the frame index. The images are rendered with BVHViewer 1.1 BID39 When motion does stagnate, it recovers relatively quickly, and the motion never diverges or freezes completely (see Figure 6). The short-term freezing could possibly be explained by "dead-times" in the training sequence, where the actor is beginning or ending a sequence which involves a rest position. Note that when training with two different datasets, as in the case of the Indian/Salsa combined network, motion borrows from both dance styles. We also demonstrate in Figure 5 that our method does not freeze even after 20,000 frames of synthesis, which is approximately 333 seconds of output. One can see a qualitative comparison of acLSTM with a basic LSTM-3LR in Figure 3, both trained on the Indian dance dataset. We find the performance of the vanilla network to be consistent with the reported in BID4 BID14 BID1 BID25, freezing at around 1000 ms. It never recovers from the motion. Our network, on the other hand, continues producing varied motion for the same time frame. We have shown the effectiveness of the acLSTM architecture to produce extended sequences of complex human motion. We believe our work demonstrates qualitative state-of-the-art in motion generation, as all previous work has focused on synthesizing relatively simple human motion for extremely short time periods. These works demonstrate motion generation up to a couple of seconds at most while acLSTM does not fail even after over 300 seconds. Though we are as of yet unable to prove indefinite stability, it seems empirically that acLSTM can generate arbitrarily long sequences. Current problems that exist include choppy motion at times, self-collision of the skeleton, and unrealistic sliding of the feet. Further developement of GAN methods, such as BID19, could in increased realism, though these models are notoriously hard to train as they often in mode collapse. Combining our technique with physically based simulation to ensure realism after synthesis is also a potential next step. Finally, it is important to study the effects of using various condition lengths during training. We begin the exploration of this topic in the appendix, but further analysis is needed. Figure 9 might imply some sort of trade off between motion change over time and short-term motion prediction error when training with different condition lengths. However, it is also possible that limiting motion magnitude on this particular dataset might correspond to lower error. Further experiments of various condition lengths on several motion styles need to be conducted to say anything meaningful about the effect. C VISUAL DIAGRAM OF AUTO-CONDITIONED LSTM

represent this text to find a much much longer article about it with say hundreds-thousands of words

Synthesize complex and extended human motions using an auto-conditioned LSTM network

Represent the TLDR for finding the corresponding scientific article:

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.