Question: Represent the scientific article to find a short TLDR

Question: Neural Style Transfer has become a popular technique for generating images of distinct artistic styles using convolutional neural networks. This recent success in image style transfer has raised the question of whether similar methods can be leveraged to alter the “style” of musical audio. In this work, we attempt long time-scale high-quality audio transfer and texture synthesis in the time-domain that captures harmonic, rhythmic, and timbral elements related to musical style, using examples that may have different lengths and musical keys. We demonstrate the ability to use randomly initialized convolutional neural networks to transfer these aspects of musical style from one piece onto another using 3 different representations of audio: the log-magnitude of the Short Time Fourier Transform (STFT), the Mel spectrogram, and the Constant-Q Transform spectrogram. We propose using these representations as a way of generating and modifying perceptually significant characteristics of musical audio content. We demonstrate each representation's shortcomings and advantages over others by carefully designing neural network structures that complement the nature of musical audio. Finally, we show that the most compelling “style” transfer examples make use of an ensemble of these representations to help capture the varying desired characteristics of audio signals. The problem we seek to explore in this paper is the transfer of artistic "style" from one musical audio example onto another. The definition and perception of an artistic style in visual art images (e.g., impressionist, pointilist, cubist) shown in Figure 1 is perhaps more straightforward than in the case musical audio. For images, a successful style transfer algorithm is capable of generating a novel image whose content information, or what is in the image, is matched as well as its stylistic information, or the artistic approach. In other words, it explores the question, "What would a rendering of scene A by artist B look like?" Figure 1: Demonstration of image style transfer courtesy of BID7.For our work, we similarly set out to develop an algorithm that explores the question, "What would it sound like if a musical piece by ensemble/artist A was performed by ensemble/artist B?" It should be noted that we do not approach the problem according to strict musicological definitions (e.g., melodic, harmonic, rhythmic, and structural elements), as one might proceed if given the musical notation of a composition. We do not presume access to the notation or any music theoretic analysis of a piece. We are instead interested in transferring the acoustic features related to harmonic, rhythmic, and timbral aspects of one musical piece onto another. Therefore, for the single instance "style" transfer algorithm we propose in this work, it is more accurate to pose the question as "What would a rendering of musical piece A (by artist A) using the harmonic and rhythmic patterns of piece B (by artist B) sound like?" In this paper, we define musical "style" transfer according to this type of audio content transformation, and will henceforth drop the use of quotation marks around "style". In texture generation, we instead ask "What would it sound like for a source musical piece to contain the same musical patterns and higher-order statistics without any of the same local, event-based information?" This can be achieved in the image or audio domain by only optimizing those terms of the loss function of a transfer algorithm associated with style, and not using any loss term associated with content. Currently, there are two types of approaches to image style transfer. The first method uses a learned generative model to manipulate the representation of the data such that it maintains its original content rendered into a new style. The second class of methods, which we investigate and apply in this paper, are concerned with synthesizing new data that matches the representations of data in a learned model in some specific way. Measuring the accuracy of such algorithms' abilities to transfer style is difficult, since most data is not able to be entirely disentangled into separate content and style components. This is especially true for musical style. There have been attempts for learning representations of musical style include the use of generative models which use a MIDI representation of audio BID14. The advantages of using this representation are the ability to focus solely on a highly understandable representation of musical information in its harmonic and rhythmic components, but lacks the ability to capture other important sonic information like timbre. Our approach utilizes many interesting findings from recent research in image style transfer. We suggest that it is possible to use the same style transfer algorithm used for images for musical audio, but best performance requires a careful selection of how content and style is represented, given the task. FIG0 shows a spectral visualization of how a style transfer contains both local, event based information from the content piece, while also having the characteristic nature of the style signal, as there is clearly more energy in the higher frequencies. However, it is important to note that despite this visualization in the log-magnitude STFT representation, the audio is ultimately synthesized in the time-domain. Original attempts at single image style transfer demonstrated successful transfer of artistic style between images through a gradient-based optimization of the convolutional map representations in the Visual Geometry Group (VGG) network BID16. This work focused on synthesizing preferred images whose layer representations in the VGG network, a convolutional neural network that rivals human performance on visual object recognition tasks, matching the likeliness of a content image while the statistics of the layer representations of a target style image are also matched. These statistics are typically represented by taking the inner product of the convolutional layer activations in order to compute the gram matrix. The ability to match each of these is expressed through separate content and style loss terms, which can both be used to optimize the pixels of a randomly initialized image. The motivation for using the gram matrix to represent style is that local information about the style image is lost, preserving only statistics about the convolutional representation over the entire image. Other work has helped explain the effectiveness of this approach by showing that single image style transfer can be re-written as a Maximum Mean Discrepancy problem with a 2nd-order polynomial distance kernel BID10. It was also shown that the need for a pre-trained network to perform style transfer was not at all necessary. In fact, the ability to recreate the content of the original image was improved when using randomly initialized weights BID6. Additionally, BID19 has shown that very shallow, untrained neural networks with as little as 1 layer are also capable of sufficiently representing artistic style in images. Ulyanov and Lebedev proposed an algorithm for the transfer of audio style using the log-magnitude STFT of audio with a single random convolutional layer BID17. After the novel log-magnitude spectrogram is generated, the Griffin-Lim algorithm BID5 is used to restore phase information before taking the inverse STFT to recover the time-domain signal. In order to generate these , only 1 convolutional layer with random, untrained weights was used with a rectified linear activation function. In order to make this model successful, 4096 different convolutional kernels must be used. Also, while spectrograms are typically thought of as F (Frequency bins) x T (Time bins) images with no color channels, this work represented them as 1xT images with F color channels. Therefore, the only convolution taking place is happening in time, while the patterns and relationships between frequencies are localized within the style loss. More recent advances in generative music models include WaveNet and SampleRNN, which also offer possibilities for style transfer BID20 BID12. These autoregressive models have successfully demonstrated the ability to generate long-term structure in audio signals. SampleRNN uses a multi-hierarchical recurrent neural network trained with Truncated Backpropogation Through Time (TBTT). WaveNet uses causal, dilated convolutions and residual skip-connections, and has the ability to be conditioned on vocal features which gives the ability to generate the same words and speech patterns with different voice characteristics, such as male or female BID21. There is also a recently proposed WaveNet Autoencoder BID2 ), a generative model for raw audio that can be used to manipulate a latent representation of an encoded sound and then regenerate it. In the previous attempt by BID17 at style transfer for audio, a simple convolutional layer with a ReLU non-linearity was used. A single 1-D convolutional layer, whose kernel has a shape that we denote as 1xKxF xN f was used. K represents the length of time bins convolved with the feature, while F and N f represent the number of frequency bins and filters. These weights are initialized using Glorot Initialization BID4. For our notation, we use x to denote the generated log-magnitude STFT, whose feature map, X is the convolved feature map over whichever dimension is being considered for style transfer or texture generation. Following this, we use s and c to denote the log-magnitude STFT representations for the style and content audio, respectively, whose feature maps are represented by S and C, respectively. The L 2 distance between the target audio and the content audio's feature maps summed over all spatial indices i and channels j is used for content loss expressed in Equation 1. DISPLAYFORM0 In representing style, it is ideal that local information about where musical events happen is lost, but information about how they happen in relation to each other is maintained. To represent the style, the inner-product, or gram matrix of the convolutional feature map is used. The inner-product of the vectorized feature maps X and S, denoted by W and G respectively, are calculated using Equations 2 and 3 for a filter i and another filter j is used to represent style. DISPLAYFORM1 The style loss, L Style, calculated as the sum of the L 2 distances between G and W over all pairs of filters i and j in N f, is given in Equation FORMULA2. DISPLAYFORM2 The total loss is represented by Equation 5, which uses parameters α and β to measure the importance of transferring each type of style. DISPLAYFORM3 All network weights are unaltered during optimization of the L. Only the log magnitude STFT representation of the target audio is adjusted. We introduce an extended version of the algorithm proposed in BID17 with different versions of two additional log-magnitude spectrogram representations described below in order to address the shortcomings of this algorithm as it applies to musical audio. In particular, BID17 characterizes and represents the timbral style of musical audio, which we can defined in this work as the short time envelope and harmonic statistics of the audio signal, but fails to capture information which is either rhythmic or harmonic. We propose using the Mel Spectrogram to better capture rhythmic information, and using the Constant Q Transform (CQT) Spectrogram in a 2-D convolutional neural network to represent harmonic style. While we use the Mel spectrogram to help represent rhythmic information, the true benefit of this representation is that the compressed channel axis allows for a deeper dilated network structure with a much longer receptive field in time. While this simple, single layer design is effective in representing short-term harmonic and timbral features of audio, its most obvious fault is its inability to effectively represent longer-term features of the audio signals, which we refer to as the rhythmic components. In order to increase the size of the kernel that spans the temporal dimension of a spectrogram, we need to also decrease the size of the filter dimension in, or the number of frequency channels. Without this, the size of our computation graph becomes very large, and the time needed for synthesis is greatly increased. We argue that rhythmic information is still preserved when compressing this frequency dimension. We propose using the same algorithm on a Mel scaled spectrogram of the log-magnitude STFT, which we refer to as the Mel spectrogram version of the signal, x M el. The Mel scale provides a mapping from the perceived Mel center frequency and the actual measured frequency, f, as shown in Equation 6 below. The mapping can be used to create a filter bank for projecting the magnitude STFT onto a perceptually optimal smaller number of channels. DISPLAYFORM0 Because the Mel spectrogram decreases spectral resolution of the STFT in a perceptually uniform manner, it has been a popular choice for state of the art neural networks trained on large corpuses of musical audio BID0 BID22. However, instead of using 2-D convolutions on the Mel spectrogram like this work, we propose treating the mel-frequency axis as a channel axis rather than a spatial axis like is done for the STFT representations. While a large number of filters in a convolutional kernel are still needed to represent style from this representation, the significantly reduced number of frequency bins means the number of our parameters in the kernel can be much smaller. In order to get a much larger receptive field of audio, we use a much longer kernel and a multi-tiered dilated non-causal convolutional structure modeled after the WaveNet auto-encoders BID2. This architecture makes use of dilated convolutions with residual, skip-connections in its encoding stage to vastly increase the receptive field. In this way, our model actually can be thought of as a generalization of this encoding model structure, but we have inserted the Mel spectrogram transformation in front of the initial layer which normally receives raw audio. While the original convolutional kernel used in BID17's implementation only had a receptive field of about 30 ms, this structure is capable of representing up to 4 seconds of audio with only 2 residual blocks for audio sampled at 22.05 kHz and N DF T = 2048. Additionally, since the dimensionality of the representation in the channel dimension is decreased by at least 2, we find that we can use a much smaller number for N f, which reduces the size of the computation graph needed for the same receptive field and decreases computation time. Following the WaveNet auto-encoder architecture, we only use dilated convolutions if more than one residual block is used, starting with a dilation rate of 2, and doubling again for each additional residual block. We compute the style loss from each layer in this neural network structure, and the content loss only using the last layer. In practice, we use between 16 (128:1 reduction) and 512 Mel filters (4:1 reduction) that are triangularly shaped and span from 0 Hz to 22.05 kHz. While the STFT and Mel spectrogram allow us to represent both short and long term musical statistics well, neither representation is capable of representing musically relevant frequency patterns, such as chords and intervals, along the frequency axis that can be shifted linearly. In order to achieve this, we need to use a 2-D convolution over a spectral representation. We have chosen the CQT spectrogram BID15 since it is capable of representing transposition in musical pitch as a simple shift along the (warped) frequency axis. In this way, an activation for a particular harmony at a note, n, should be close in value to the same harmonic structure being present at any other note within +/-6 semitones. Similarly to the Mel spectrogram, it has often been chosen as a representation for neural networks for this reason. While in some ways the CQT spectrogram is worse than the Mel spectrogram in terms of perceptual reconstruction quality due to the frequency scaling in lower frequencies, it is the most natural representation for 2-D convolutional kernels to use to represent joint time-frequency spatial features. We implement the "pseudo-CQT", which is obtained by projecting the magnitude STFT signal onto a wavelet filter bank with Constant Q frequency scaling, rather than the recursive sub-sampling method described in BID15, which has the benefit of modeling the transformation as a fully differentiable operation on the raw time-domain audio. In order to achieve key invariance for musical content, we use a total of 86 bins with 12 bins per octave spanning from the notes C1 to D8, and use max-pooling along the convolved frequency axis if key-invariance is desired. This allows features from the target audio to match the features of the content and style audio from multiple keys. As the width of the pooling increases, the representation becomes more key-invariant, and the distortion of the content representation increases. This filter bank is similarly logarithmic to the Mel filter bank in the higher frequency range (> 1kHz), but it oversamples the lower frequencies in comparison.3.3 PARALLEL ARCHITECTURE AND TIME DOMAIN SYNTHESIS We generalize the formulation described in section 3.1 for the content and style loss to yield 6 total possible loss terms (content and style for all 3 networks) to be used during optimization: DISPLAYFORM0 For each content loss, use the L 2 loss of the final layers' representations. For the style loss terms, we use the sum of all L 2 loss of the gram matrix representations of each layer in the network as described in Section 3.1. For the CQT network representations, the inner-product of both the temporal and frequency axes is used since both frequency and time invariance is desired. In practice, we make use of all possible terms, using up to 5 at once. Since there are up to 5 different objectives in our proposed solution, it can be difficult to discover the optimal weighting for a style transfer example. In order to alleviate this process, we propose initializing each loss term's coefficient such that the magnitude of the gradient is 1 for all loss terms as shown in Equation 7, inspired by BID10. We then control the scaling of the then normalized loss terms using the parameter, Γ T ype,N et, as shown in Equation 8. DISPLAYFORM1 The objective function to be minimized is the sum of all of these losses scaled by β T ype,N et as expressed in Equation 9. DISPLAYFORM2 The previous method computes the log-magnitude STFT as a pre-processing step and, after optimizing the , used the Griffin-Lim algorithm BID5 to reconstruct phase. We, instead, propose a method for estimating phase simultaneously with the optimization of the STFT representation. This transformation is fully differentiable to the raw time domain audio signal, meaning if overlapping time windows are used, phase information can be approximated. We use gradient descent based optimization algorithm Limited Memory BFGS (L-BFGS-B) for its superior performance in non-linear optimization problems including image style transfer. Recent work has shown that BFGS is not only capable of reconstructing phase in time-domain signals from magnitude STFT representations, but it converges faster and can reconstruct phase on some signals on which Griffin-Lim fails BID1. Because of this, we choose to model the transformation from the time-domain target input to the log-magnitude STFT as well as the Mel and CQT representations extracted as a projection of the target audio onto different bases as symbolic combinations of differentiable convolutional and dense layers in our computation graph. This allows us to optimize the time-domain audio from each spectral representation network's gradient in one stage without suffering major phase distortion. Another advantage of optimizing the target audio directly in the time domain is that it allows also for neural representations of the time-domain audio to be optimized simultaneously. We see this as being outside of the scope of the work being presented here, however. While each of these representations has complementary advantages and disadvantages, we can still use all 3 representations in parallel neural networks to capitalize upon the advantages from each. This is entirely possible through the ensemble representation of each network shown in FIG1 below. Prior work in style transfer proposed that higher layer neural representations of images captured the content more than style, where content is any information relevant to the classification task. We seek to create a similarly meaningful content representation, which maintains general information about musical events, without specifics like key and timbre. Key-invariance is desired since modern popular music is often composed without an emphasis on the specific key, and instead the harmony is commonly defined as its relation to the "one", which in Western music can be any of 12 musical keys. For these reasons, we argue that key-invariance is a crucial feature for a representation of musical content. We propose and test 3 methods for examining invariance to musical key between the content and style audio sources. For the first, we suggest using a Mel spectrogram content representation with only a few channels. This has the effect of taking a wide-band average of the audio signal, capturing the envelope of the audio signal. This type of representation captures only the rhythmic hits and the overall phonemes of words being sung. While prominent rhythmic and timbral shape information is preserved, all information about key is lost since the signal is being averaged over large regions of frequency. In For the second key-invariant content representation, we wish to use the convolved 2-D feature map of the CQT spectrogram. We choose a convolutional kernel that is 11x11, meaning it spans just under one octave at each stride. We use a max-pooling layer in order to make this content representation keyinvariant, and we use max-pooling along the convolved frequency axis of the convolutional feature map. This representation ensures that content that has been shifted in key, or shifted along the warped frequency axis of the CQT spectrogram, will yield a similar representation as the non-shifted version of the same signal. However, it is important to note that this max-pooling layer is only beneficial for making L Content,CQT key-invariant, since the L Style,CQT uses the inner product of the spatial dimension of the data, making it already key-invariant. In comparison with the key-invariant Mel spectrogram representation, this representation better preserves melodic and harmonic information without maintaining local key information. In practice we use a stride of 2 for the pooling, however, a greater stride could be used for increased invariance at the cost of greater information loss. It is important to note that pooling occurs on the convolutional layer versions of the representation, not the raw CQT itself. Pooling in this fashion would destroy harmonic information, while pooling on extracted patterns spanning enough frequency allows for these same patterns (i.e. a major third interval) to be placed at any fundamental frequency within the width of semi-tones defined by the pooling size. Finally, for the best approximation, we recommend combining both of these loss terms to encapsulate the information from each key-invariant representation. Previous work in image style transfer has shown that using instance normalization layers can improve quality of the synthesized images. Unfortunately, when we attempted to implement these for audio, these layers also seemed to introduce a noticeable amount of noise to the target audio. Instead, we use newly proposed self-normalizing non-linear activation functions, SeLUs BID8, in place of ReLUs BID13 since they have both desirable non-linear properties of activation functions as well as the property of allowing network activations to have 0 mean and unit variance with LeCun Normal initialization BID9. We find that using this activation function increases the quality of the synthesized audio, reduces convergence time, and makes finding an optimal weighting for the style and content terms more consistent from example to example. Since we don't use any type of dense neural network layers in our proposed networks, we demonstrate that it is possible to use different lengths of content and style audio with the same convolutional kernels and still obtain a valid style loss. To achieve this, we simply divide by the size of the convolutional feature map being used prior to computing the difference in the style loss calculation. We propose 3 experiments for showing the ability for our algorithm to improve the transfer of musical style. First, we show how using the Mel spectrogram representation of style is able to better capture the musical statistics of the audio through texture generation. We do so by examining what happens when we generate musical textures, meaning only style loss with no content. We also examine how the combined representation of the Mel and CQT representations of content offer a frequency invariant representation. Finally, we examine the quality of our algorithm for a variety of musical style transfer examples using both quantitative and qualitative assessment. For hyper-parameters, we find that using 4096 filters for the STFT network, 1025 for the Mel network, and 256 for the CQT network in each convolutional layer to be sufficient. We use kernel sizes of 1x11 (~25 ms), 1x50 (~1 second at first layer), and 11x11 (11 semitones x~25 ms) for the STFT, Mel, and CQT networks respectively as well. However, for residual layers in the Mel network, we use a 1x25 kernel. Since we use SELUs, we also use LeCun Normal weight initialization BID9.All experiments were conducted using a Tensorflow implementation of the ensemble model described in the previous section. We have released code used for the experiments. 1 Additionally, we also have a collection of musical style transfer examples, including all of the examples used for the shown below. While textures synthesized using prior methods exhibit a lack of rhythmic structure, we show the use of residual dilated non-causal convolutions on the Mel spectrogram works well for achieving a large receptive field and better capturing rhythmic statistics. In order to measure the rhythmic style similarity of a texture to its source audio, we compute the Kullback-Leibler (KL)-divergence of the Inter-Onset Interval Length Distributions. we detect onsets using Librosa's onset detector, which picks peaks based on the differences of energy in adjacent time frames of a spectrogram BID11. While increasing the receptive field to a much longer length improves this measure of rhythmic similarity, it is important to show that this is not introducing the effect of copying sections of the original source audio. In order to verify that this is not occurring, we also show that the maximum cross-correlation value between the time-domain audio waveforms are not significantly affected by the length of this field. We denote max(Φ XY) as the maximum cross-correlation value between two signals, X and Y. This helps justify that the source and texture are still significantly different, so that the texture isn't simply a distorted duplicate of the source. Additionally, we show that the mean local autocorrelation has more consistent hierarchical rhythmic structure at different lag times. Figure 4 summarizes these . In order to test the quality of key-invariance we suggest that it must be possible to use songs in two different keys for style and content, and be able to reconstruct the same content in the key of the style audio. We synthesized 13 different transpositions from a MIDI score of 4 second clips from "The Star Spangled Banner" and performed style transfer experiments where the original key version is used as Figure 4: Columns 1 and 3: comparison of inter-onset lengths distribution and KL divergence from the source distribution for a texture generation example as the effective receptive field increases in time. Columns 2 and 4: mean local autocorrelation plots showing the increase in hierarchical rhythmic structure of the audio without any significant increase in the maximum cross-correlation value, max(Φ XY). content, and the versions which are transposed +/-6 semitones are used for style. We used both the mean squared error of the log-magnitude STFT representation of the and the style used as a metric for content key-invariance. We choose this metric because the log-magnitude STFT contains most information about original time-domain audio signal, and we argue that being able to reconstruct the exact style signal should be possible if the content and style are the same, but in different keys. FIG3 shows the full for all experiments. Our confirm that changing key between style content has less of an effect on our proposed key-invariant content representations. The confirm that for all cases where the content and style pieces are in different keys, all of the proposed key-invariant representations have lower error than the normal STFT representations. While there is no clear best representation of these three proposed versions according to this quantitative metric, we observe that using both the CQT and M el representations captures different kinds of key-invariant information and combining these representations yields the best sounding . Samples from this experiment are included in the supplement. We tested the full style transfer algorithm with different combinations of loss terms and hyperparameters for a diverse set of examples. We found that there is not one set of loss terms that works best for all content-style transfer pairs, as the best set of loss terms is usually dependent on the nature of the specific style transfer task. However, we are able to demonstrate specific cases where using only the STFT loss terms fails, and significant improvement is achieved with the introduction of the other objectives based on perceptual audio representations. All examples use content and style with tempos within 10 beats per minute (bpm) of each other. We summarize our findings for a variety of loss content and style representation formulations in Table 1. Table 1: Summary of the effects of using different possible loss term formulations. We find that using both of our proposed content and style loss formulations improves the example quality from the simple STFT architecture proposed in BID17 We notice that using Mel spectrogram representations for achieving a larger receptive field in time greatly helps transfer style in certain cases where the long-term nature of the style audio is complex, and important to recognizing the nature of the musical style. Using both Mel and CQT representations for content greatly helps for cases where the nature of the content and style audio are very different from each other, even in cases where the content and style are in the same key. The abstracted content representation is better able to re-implement content with the musical attributes of the style audio without simply laying 2 different sounding examples over each other. This increase in performance is best displayed in examples of singing style transfer. Using a higher number of Mel channels and residual blocks will give the behavior similar to a "mash-up" of 2 audio examples, where portions of the style audio are placed over the content audio. Simply reducing the number of Mel channels eliminates this behavior so that the L Style,ST F T has high resolution frequency information over a short time span, representing the timbral and short-time harmonic features only, and L Style,M el uses a much longer time span with low frequency resolution, representing long-term musical attributes like rhythmic structure. When we use the Mel spectrogram content representation, we notice that the wide-frequency band nature of the synthesized audio sometimes has an undesirable, "breathy" quality to it. This problem is most evident when there are vocals. In order to restrain this effect, we add the L 1 penalty to the log-magnitude STFT to enforce the synthesized audio to have a sparse spectrum. We find that this helps eliminate the evenly distributed nature of the content part of the synthesized audio. We introduce several improvements for performing musical style transfer on raw audio through the utilization of multiple audio representations. Our contributions can be summarized as follows: First, we have demonstrated that using additional representations of Mel and CQT spectrograms with accompanying neural structure improve in many cases the capture of musically meaningful style information. Secondly, we have proposed a novel, key-invariant content representation for musical audio. Finally we have shown that despite using log-magnitude spectrograms to capture the content and style information, we are still able to synthesize a target audio waveform in the time domain using the backpropogation of the STFT.While our proposed content representations work for audio in different keys, there still is no representation for tempo invariance. Other future work may include using learned generative models to perform musical style transfer and trying to perform style transfer entirely in the time-domain. This or the use of complex weights may be able to help improve representation of phase information in neural representations.

We present a long time-scale musical audio style transfer algorithm which synthesizes audio in the time-domain, but uses Time-Frequency representations of audio.

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: To communicate with new partners in new contexts, humans rapidly form new linguistic conventions. Recent language models trained with deep neural networks are able to comprehend and produce the existing conventions present in their training data, but are not able to flexibly and interactively adapt those conventions on the fly as humans do. We introduce a repeated reference task as a benchmark for models of adaptation in communication and propose a regularized continual learning framework that allows an artificial agent initialized with a generic language model to more accurately and efficiently understand their partner over time. We evaluate this framework through simulations on COCO and in real-time reference game experiments with human partners. Linguistic communication depends critically on shared knowledge about the meanings of words BID9. However, the real-world demands of communication often require speakers and listeners to go beyond dictionary meanings to understand one another BID0 BID15. The social world continually presents new communicative challenges, and agents must continually coordinate on new meanings to meet them. For example, consider a nurse visiting a bed-ridden patient in a cluttered home. The first time they ask the nurse to retrieve a particular medication, the patient must painstakingly refer to unfamiliar pills, e.g. "the vasoprex-tecnoblek meds for my blood pressure, in a small bluish bottle, on the bookcase in my bathroom." After a week of care, however, they may just ask for their "Vasotec."This type of flexible language use poses a challenge for models of language in machine learning. Approaches based on deep neural networks typically learn a monolithic meaning function during training, with fixed weights during use. For an in-home robot to communicate as flexibly and efficiently with patients as a human nurse, it must be equipped with a continual learning mechanism. Such a mechanism would present two specific advantages for interaction and communication applications. First, to the extent that current models have difficulty communicating in a new setting, an adaptive approach can quickly improve performance on the relevant subset of language. Second, for human-robot contexts, an adaptive model enables speakers to communicate more efficiently as they build up common ground, remaining understandable while expending significantly fewer words as humans naturally do BID1.In this paper, we introduce a benchmark communication task and general continual learning framework for transforming neural language models into adaptive models that can be deployed in real-time interactions with other agents. Our key insight is that through continual interactions with the same partner in a shared context, an adaptive listener can more effectively communicate with its partner FIG0.We are motivated by hierarchical Bayesian approaches to task-specific adaptation. Our approach integrates two core components: (i) a loss function combining speaker and listener information, and (ii) a regularization scheme for fine-tuning model weights without overfitting. We begin by recasting communication as a multi-task problem for meta-learning. Each context and communicative partner can be regarded as a related but distinct task making its own demands on the agent's language model. To be effective across many such tasks, a communicative agent must both have a good prior representation they can use to understand novel partners and contexts, and have a mechanism to rapidly update this representation from a small number of interactions. As a benchmark for studying this problem, we introduce the repeated reference game task FIG0, which has been widely used in cognitive science to study partner-specific adaptation in communication BID8 BID1 BID18. In this task, a speaker agent and a listener agent are shown a context of images, C, and must collaborate on how to refer to them. On each trial, one of these images is privately designated as the target object o for the speaker. The speaker thus takes the pair (o, C) as input and returns an utterance u that will allow the listener to select the target. The listener agent then takes (u, C) as input and returns a softmax probability for each image, which it uses to make a selection. Both agents then receive feedback about the listener's response and the identity of the target. Critically, the sequence of trials is constructed so that each image repeatedly appears as the target, allowing us to evaluate how communication about each image changes over time. Before formalizing our algorithm as a generic update rule for neural networks, we describe the theoretical Bayesian foundations of our approach. At the core of any communication model is a notion of the semantics of language, which supplies the relationship between utterances and states of the world. Under a Bayesian approach, this representation is probabilistic: we represent some uncertainty over meanings. In a hierarchical Bayesian model, this uncertainty is structured over different partners and contexts. At the highest level of the hierarchy is a task-general variable Θ which parameterizes the agent's task-specific prior expectations P (θ i |Θ), where θ i represents the semantics used by a novel partner i. Given observations D i from communicative interactions in that context, an agent can update their task-specific model using Bayes rule: DISPLAYFORM0 The Bayesian formulation thus decomposes the problem of task-specific adaptation into two terms, a prior term P (θ i |Θ) and a likelihood term P (D i |θ i). The prior captures the idea that different language tasks share some task-general structure in common: in the absence of strong information about usage departing from this common structure, the agent ought to be regularized toward their task-general knowledge. The likelihood term accounts for needed deviations from general knowledge due to evidence from the current situation. The form of the likelihood depends on the task at hand. For our benchmark communication task, D i = {(u, o) t } contains paired observations of utterances u and their objects of reference o at times t. These data can be viewed from the point of view of a speaker (generating u given o) or a listener (choosing o from a context of options, given u) BID14 BID6. A speaker model 1 uses its task-specific semantics θ i to sample utterances u proportional to how well they apply to o: DISPLAYFORM1 A listener can be modeled as inverting this speaker model to evaluate how well an utterance u describes each object o relative to the others in a context C of objects by normalizing BID3 BID16; BID11: DISPLAYFORM2 Because these views of the data D i provide complementary statistical information about the task-specific semantics θ i, we will combine them in our loss. There is a deep theoretical connection between the hierarchical Bayesian framework presented in the previous section and recent deep learning approaches to multi-task learning BID12 BID5 BID7. Given a task-general initialization, regularized gradient descent on a particular task is equivalent to conditioning on new data under a Bayesian prior. We exploit this connection to propose an online continual learning scheme for a neural listener model that can adapt to a human speaker in our challenging referential communication task. Concretely, we consider an image-captioning network that combines a convolutional visual encoder (ResNet-152) with an LSTM decoder BID17. The LSTM takes a 300-dimensional embedding as input for each word in an utterance and its output is then linearly projected back to a softmax distribution over the vocabulary size. To pass the visual feature vector computed by the encoder into the decoder, we replaced the final layer of ResNet with a fullyconnected adapter layer. This layer was jointly pre-trained with the decoder on the COCO training set and then frozen, Algorithm 1 Update step for adaptive language model Input: θ t: weights at time t Output: θ t+1: updated weights Data: (u t, o t): observed utterance and object at time t for step do sample augmented batch of sub-utterances u ∼ P(u) update θ t ← θ t + β∇[P (u|o) + P (o|u) + reg(o, u)] end for leaving only the decoder weights (i.e. word embeddings, LSTM, and linear output layer) to be learned in an online fashion. Upon observing each utterance-object data point in the current task, we take a small number of gradient steps fine-tuning these weights to better account for the speaker's usage (see Algorithm 1). We consider several loss terms and techniques to do so. Speaker and listener likelihood. The primary signal available for adaptation is the (log-) probability of the new data under speaker and listener likelihoods given in Eqns. 2-3. Our speaker likelihood serves to make the observed utterance more likely for the target in isolation, while our listener likelihood makes it more likely relative to other objects in context. The speaker and listener likelihoods can be computed directly from the neural captioning model, as shown in FIG0, where the probability of each word is given by the softmax decoder output conditioned on the sentence so far. Regularization. We introduce two kinds of regularization terms to approximate the Bayesian prior on task-specific learning. First, rather than directly regularizing weights, a global KL regularization term minimizes the divergence between the captioning model's output probabilities before and after fine-tuning BID19 BID4. Since the support for our distribution of captions is infinite, we approximate the divergence incrementally by expanding from the maximum a posteriori (MAP) word at each step according to P, where P represents the model at initialization and Q t represents the model at time t. This loss is then averaged across random images from the full domain O, not just those in context: DISPLAYFORM0 where we denote the word at position i by w i and terminate after reaching L, the length of the MAP caption. A second form of regularization we consider is local rehearsal: we sum the likelihood over previous observations (u, o) τ from the same partner to prevent overfitting to the most recent observation. Finally, we examine listener variants of both forms of regularization by using the listener likelihood instead of the speaker likelihood. For example, we compute the listener KL regularization by comparing the initial listener distribution over the objects in context o ∈ C with the fine-tuned model's distribution: D KL (P (o|u)||Q t (o|u)). We anneal the weight on the listener regularization terms over time while reverse-annealing the listener likelihood. Data augmentation. A final component of our approach is a data augmentation step on the new utterance u. Ideally, an adaptive agent should learn that words and sub-phrases contained in the observed utterance are compositionally responsible for its meaning. We thus derive a small training dataset D(u) from u; for simplicity, we take the (ordered) powerset D(u) = P(u) of all sub-utterances. To evaluate our model, we implemented a repeated reference game using images from the validation set of COCO BID10 as the targets of reference. To construct challenging contexts C, we used our pre-trained visual encoder to find sets of highly similar images. We extracted feature vectors for each image, partitioned the images into 100 groups using a k-means algorithm, sampled one image from each cluster, and took its 3 nearest neighbors in feature space, yielding 100 unique contexts of 4 images each 3. We first investigated the baseline performance of human speakers and listeners. We recruited 113 participants from Amazon Mechanical Turk and automatically paired them into an interactive environment with a chatbox. For each of these 56 pairs, we sampled a context and constructed a sequence of 24 trials structured into 6 repetition blocks, where each of the 4 images appeared as the target once per block. We prevented the same target appearing twice in a row and scrambled the order of the images on each player's screen on each trial. We found that pairs of humans were remarkably accurate at this task, with performance near ceiling on every round. At the same time, they grew increasingly efficient in their communication: the utterance length decreased from an average of 7 words per image on the first repetition to only 3 words on the last. A mixed-effects regression with random slopes and intercepts accounting for variability at the pairand context-level found a significant decrease in utterance length across repetitions, t = −5.8, p < 0.001 (FIG2 . Next, we evaluated how our adaptive listener performed in real-time interaction with human speakers. We recruited 45 additional participants from Amazon Mechanical Turk who were told they would be paired with an artificial agent learning how they talk. This task was identical to the one performed by humans, except participants were only allowed to enter a single message through the chatbox on each trial. This message was sent to a GPU where the model weights from the previous trial were loaded, used to generate a response, and updated in real-time for the next round. The approximate latency for the model to respond was 6-8s. We used a batch size of 8, learning rate of 0.0005, and took 8 gradient steps after each trial. For our loss objective, we used a linear combination of all speaker and listener likelihood losses and regularization terms. We found that a listener based on a pre-trained neural captioning modelthe initialization for our adapting model-performs much less accurately than humans due to the challenging nature of the reference task. Yet our model rapidly improves in accuracy as it coordinates on appropriate meanings with human speakers. Similarly, while speakers did not simplify their utterances to the same extent as they did with other humans, perhaps due to early feedback about errors, they nonetheless became significantly more efficient over time, b = −19, t = −5 (see FIG2). We proceed to a series of lesion analyses that analyze the role played by each component of our approach. Fine-tuning repeatedly on a small number of data points presents a clear risk of catastrophic forgetting BID13, losing our ability to produce or understand utterances for other images. Our KL regularization term (Eqn. 4) was intended to play the same role as a Bayesian prior, preventing catastrophic forgetting by tethering task-specific behavior to the task-general model. To test the effectiveness of this term, we examined the likelihood of different captions before and after adaptation to the human baseline utterances. First, we sampled a random set of images from COCO that were not used in our experiment as control images, and used the initialized state of the LSTM to greedily generate a caption for each. We also generated initial captions for the target objects in context. We recorded the likelihood of all of these sampled captions under the model at the beginning and at each step of adaptation until the final round. Finally, we greedily generated an utterance for each target at the end and retrospectively evaluated its likelihood at earlier states. These likelihood curves are shown with and without speaker KL regularization in FIG2. The final caption becomes more likely in both cases; without the KL term, the initial captions for both targets and unrelated controls are (catastrophically) lost. We next simulated our adaptive agent's performance understanding utterances from the human baseline under lesioned losses FIG2. We found that rehearsal on previous rounds had the largest qualitative benefit, allowing for faster adaptation on early rounds, while data augmentation and the listener terms provided small boosts later in the game. Compared to a non-adapting baseline, however, even a simple loss only containing the speaker likelihood and speaker KL regularization performed better over time-successfully adapting to human language use. Human language use is flexible, continuously adapting to the needs of the current situation. In this paper, we introduced a challenging repeated reference game benchmark for artificial agents, which requires such adaptability to succeed. We proposed a continual learning approach that forms context-specific conventions by adapting general-purpose semantic knowledge. Even when models based on generalpurpose knowledge perform poorly, our approach allows human speakers working with adapted variants of such models to become more accurate and more efficient over time.

Represent the short summary for finding the corresponding scientific article (long).

We propose a repeated reference benchmark task and a regularized continual learning approach for adaptive communication with humans in unfamiliar domains

We learn an unsupervised learning algorithm that produces useful representations from a set of supervised tasks. At test-time, we apply this algorithm to new tasks without any supervision and show performance comparable to a VAE.


Question: Represent the next text

Question: Traditional set prediction models can struggle with simple datasets due to an issue we call the responsibility problem. We introduce a pooling method for sets of feature vectors based on sorting features across elements of the set. This can be used to construct a permutation-equivariant auto-encoder that avoids this responsibility problem. On a toy dataset of polygons and a set version of MNIST, we show that such an auto-encoder produces considerably better reconstructions and representations. Replacing the pooling function in existing set encoders with FSPool improves accuracy and convergence speed on a variety of datasets. Consider the following task: you have a dataset wherein each datapoint is a set of 2-d points that form the vertices of a regular polygon, and the goal is to learn an auto-encoder on this dataset. The only variable is the rotation of this polygon around the origin, with the number of points, size, and centre of it fixed. Because the inputs and outputs are sets, this problem has some unique challenges. Encoder: This turns the set of points into a latent space. The order of the elements in the set is irrelevant, so the feature vector the encoder produces should be invariant to permutations of the elements in the set. While there has been recent progress on learning such functions , they compress a set of any size down to a single feature vector in one step. This can be a significant bottleneck in what these functions can represent efficiently, particularly when relations between elements of the set need to be modeled (; b). Decoder: This turns the latent space back into a set. The elements in the target set have an arbitrary order, so a standard reconstruction loss cannot be used naïvely -the decoder would have to somehow output the elements in the same arbitrary order. Methods like those in therefore use an assignment mechanism to match up elements (section 2), after which a usual reconstruction loss can be computed. Surprisingly, their model is still unable to solve the polygon reconstruction task with close-to-zero reconstruction error, despite the apparent simplicity of the dataset. In this paper, we introduce a set pooling method for neural networks that addresses both the encoding bottleneck issue and the decoding failure issue. We make the following contributions: 1. We identify the responsibility problem (section 3). This is a fundamental issue with existing set prediction models that has not been considered in the literature before, explaining why these models struggle to model even the simple polygon dataset. 2. We introduce FSPOOL: a differentiable, sorting-based pooling method for variable-size sets (section 4). By using our pooling in the encoder of a set auto-encoder and inverting the sorting in the decoder, we can train it with the usual MSE loss for reconstruction without the need for an assignment-based loss. This avoids the responsibility problem. 3. We show that our auto-encoder can learn polygon reconstructions with close-to-zero error, which is not possible with existing set auto-encoders (subsection 6.1). This benefit transfers over to a set version of MNIST, where the quality of reconstruction and learned representation is improved (subsection 6.2). In further classification experiments on CLEVR (subsection 6.3) and several graph classification datasets (subsection 6.4), using FSPool in a set encoder improves over many non-trivial baselines. Lastly, we show that combining FSPool with Relation Networks significantly improves over standard Relation Networks in a model that heavily relies on the quality of the representation (subsection 6.5). The problem with predicting sets is that the output order of the elements is arbitrary, so computing an elementwise mean squared error does not make sense; there is no guarantee that the elements in the target set happen to be in the same order as they were generated. The existing solution around this problem is an assignment-based loss, which assigns each predicted element to its "closest" neighbour in the target set first, after which a traditional pairwise loss can be computed. We have a predicted setŶ with feature vectors as elements and a ground-truth set Y, and we want to measure how different the two sets are. These sets can be represented as matrices with the feature vectors placed in the columns in some arbitrary order, soŶ = [ŷ,...,ŷ (n) ] and Y = [y,..., y (n) ] with n as the set size (columns) and d as the number of features per element (rows). In this work, we assume that these two sets have the same size. The usual way to produceŶ is with a multi-layer perceptron (MLP) that has d × n outputs. Linear assignment One way to do this assignment is to find a linear assignment that minimises the total loss, which can be solved with the Hungarian algorithm in O(n 3) time. With Π as the space of all n-length permutations: Chamfer loss Alternatively, we can assign each element directly to the closest element in the target set. To ensure that all points in the target set are covered, a term is added to the loss wherein each element in the target set is also assigned to the closest element in the predicted set. This has O(n 2) time complexity and can be run efficiently on GPUs. Both of these losses are examples of permutation-invariant functions: the loss is the same regardless of how the columns of Y andŶ are permuted. It turns out that standard neural networks struggle with modeling symmetries that arise because there are n! different list representations of the same set, which we highlight here with an example. Suppose we want to train an auto-encoder on our polygon dataset and have a square (so a set of 4 points with the x-y coordinates as features) with some arbitrary initial rotation (see Figure 1). Each pair in the 8 outputs of the MLP decoder is responsible for producing one of the points in this square. We mark each such pair with a different colour in the figure. If we rotate the square (top left in figure) by 90 degrees (top right in figure), we simply permute the elements within the set. They are the same set, so they also encode to the same latent representation • • − Figure 1: Discontinuity (red arrow) when rotating the set of points. The coloured points denote which output of the network is responsible for which point. In the top path, the set rotated by 90 • is the same set (exactly the same shape before and after rotation) and encodes to the same feature vector, so the output responsibility (colouring) must be the same too. In this example, after 30 • and a further small clockwise rotation by, the point that each output pair is responsible for has to suddenly change. and decode to the same list representation. This means that each output is still responsible for producing the point at the same position after the rotation, i.e. the dark red output is still responsible for the top left point, the light red output is responsible for the top right point, etc. However, this also means that at some point during that 90 degree rotation (bottom path in figure), there must exist a discontinuous jump (red arrow in figure) in how the outputs are assigned. We know that the 90 degree rotation must start and end with the top left point being produced by the dark red output. Thus, we know that there is a rotation where all the outputs must simultaneously change which point they are responsible for, so that completing the rotation in the top left point being produced by the dark red output. Even though we change the set continuously, the list representation (MLP or RNN outputs) must change discontinuously. This is a challenge for neural networks to learn, since they can typically only model functions without discontinuous jumps. As we increase the number of vertices in the polygon (number of set elements), it must learn an increasing frequency of situations where all the outputs must discontinuously change at once, which becomes very difficult to model. Our experiment in subsection 6.1 confirms this. This example highlights a more general issue: whenever there are at least two set elements that can be smoothly interchanged, these discontinuities arise. We show this more formally in Appendix A. For example, the set of bounding boxes in object detection can be interchanged in much the same way as the points of our square here. An MLP or RNN that tries to generate these (like in ;) must handle which of its outputs is responsible for what element in a discontinuous way. Note that traditional object detectors like Faster R-CNN do not have this responsibility problem, because they do not treat object detection as a proper set prediction task with their anchor-based approach. The main idea behind our pooling method is simple: sorting each feature across the elements of the set and performing a weighted sum. The numerical sorting ensures the property of permutationinvariance. The difficulty lies in how to determine the weights for the weighted sum in a way that works for variable-sized sets. A key insight for auto-encoding is that we can store the permutation that the sorting applies in the encoder and apply the inverse of that permutation in the decoder. This allows the model to restore the arbitrary order of the set element so that it no longer needs an assignment-based loss for training. This avoids the problem in Figure 1, because rotating the square by 90 • also permutes the outputs of the network accordingly. Thus, there is no longer a discontinuity in the outputs during this rotation. In other words, we make the auto-encoder permutation-equivariant: permuting the input set also permutes the neural network's output in the same way. We describe the model for the simplest case of encoding fixed-size sets in subsection 4.1, extend it to variable-sized sets in subsection 4.2, then discuss how to use this in an auto-encoder in subsection 4.3. Figure 2: Overview of our FSPOOL model for variable-sized sets. In this example, the weights define piecewise linear functions with two pieces. The four dots on each line correspond to the positions where f is evaluated for a set of size four. We are given a set of n feature vectors where each x (i) is a column vector of dimension d placed in some arbitrary order in the columns of X ∈ R d×n. From this, the goal is to produce a single feature vector in a way that is invariant to permutation of the columns in the matrix. We first sort each of the d features across the elements of the set by numerically sorting within the rows of X to obtain the matrix of sorted features X: where X i,: is the ith row of X and SORT(·) sorts a vector in descending order. While this may appear strange since the columns of X no longer correspond to individual elements of the set, there are good reasons for this. A transformation (such as with an MLP) prior to the pooling can ensure that the features being sorted are mostly independent so that little information is lost by treating the features independently. Also, if we were to sort whole elements by one feature, there would be discontinuities whenever two elements swap order. This problem is avoided by our featurewise sorting. Efficient parallel implementations of SORT are available in Deep Learning frameworks such as PyTorch, which uses a bitonic sort (O(log 2 n) parallel time, O(n log 2 n) comparisons). While the permutation that the sorting applies is not differentiable, gradients can still be propagated pathwise according to this permutation in a similar way as for max pooling. Then, we apply a learnable weight matrix W ∈ R d×n to X by elementwise multiplying and summing over the columns (row-wise dot products). y ∈ R d is the final pooled representation of X. The weight vector allows different weightings of different ranks and is similar in spirit to the parametric version of the gather step in Gather-Excite . This is a generalisation of both max and sum pooling, since max pooling can be obtained with the weight vector [1, 0, . . ., 0] and sum pooling can be obtained with the 1 vector. Thus, it is also a maximally powerful pooling method for multi-sets while being potentially more flexible in what it can represent. When the size n of sets can vary, our previous weight matrix can no longer have a fixed number of columns. To deal with this, we define a continuous version of the weight vector in each row: we use a fixed number of weights to parametrise a piecewise linear function f: → R, also known as calibrator function . For a set of size three, this function would be evaluated at 0, 0.5, and 1 to determine the three weights for the weighted sum. For a set of size four, it would be evaluated at 0, 1/3, 2/3, and 1. This decouples the number of columns in the weight matrix from the set size that it processes, which allows it to be used for variable-sized sets. To parametrise a piecewise linear function f, we have a weight vectorw ∈ R k where k − 1 is the number of pieces defined by the k points. With the ratio r ∈, The max(·) term selects the two nearest points to r and linearly interpolates them. For example, if k = 3, choosing r ∈ [0, 0.5] interpolates between the first two points in the weight vector with (1 − 2r)w 1 + 2rw 2. We have a differentw for each of the d features and place them in the rows of a weight matrix W ∈ R d×k, which no longer depends on n. Using these rows with f to determine the weights: y is now the pooled representation with a potentially varying set size n as input. When n = k, this reduces back to Equation 4. For most experiments, we simply set k = 20 without tuning it. To create an auto-encoder, we need a decoder that turns the latent space back into a set. Analogously to image auto-encoders, we want this decoder to roughly perform the operations of the encoder in reverse. The FSPool in the encoder has two parts: sorting the features, and pooling the features. Thus, the FSUnpool version should "unpool" the features, and "unsort" the features. For the former, we define an unpooling version of Equation 6 that distributes information from one feature vector to a variable-size list of feature vectors. For the latter, the idea is to store the permutation of the sorting from the encoder and use the inverse of it in the decoder to unsort it. This allows the auto-encoder to restore the original ordering of set elements, which makes it permutation-equivariant. With y ∈ R d as the vector to be unpooled, we define the unpooling similarly to Equation 6 as In the non-autoencoder setting, the lack of differentiability of the permutation is not a problem due to the pathwise differentiability. However, in the auto-encoder setting we make use of the permutation in the decoder. While gradients can still be propagated through it, it introduces discontinuities whenever the sorting order in the encoder for a set changes, which we empirically observed to be a problem. To avoid this issue, we need the permutation that the sort produces to be differentiable. To achieve this, we use the recently proposed sorting networks , which is a continuous relaxation of numerical sorting. This gives us a differentiable approximation of a permutation matrix.., d} for each of the d features, which we can use in the decoder while still keeping the model fully differentiable. It comes with the trade-off of increased computation costs with O(n 2) time and space complexity, so we only use the relaxed sorting in the auto-encoder setting. It is possible to decay the temperature of the relaxed sort throughout training to 0, which allows the more efficient traditional sorting algorithm to be used at inference time. Lastly, we can use the inverse of the permutation from the encoder to restore the original order. where P T i permutes the elements of the ith row in X. Because the permutation is stored and used in the decoder, this makes our auto-encoder similar to a U-net architecture since it is possible for the network to skip the small latent space. Typically we find that this only starts to become a problem when d is too big, in which case it is possible to only use a subset of the P i in the decoder to counteract this. We are proposing a differentiable function that maps a set of feature vectors to a single feature vector. This has been studied in many works such as Deep Sets and PointNet , with universal approximation theorems being proven. In our notation, the Deep Sets model is g(j h(X :,j)) where h: Since this is O(n) in the set size n, it is clear that while it may be able to approximate any set function, problems that depend on higher-order interactions between different elements of the set will be difficult to model aside from pure memorisation. This explains the success of relation networks (RN), which simply perform this sum over all pairs of elements, and has been extended to higher orders by. Our work proposes an alternative operator to the sum that is intended to allow some relations between elements to be modeled through the sorting, while not incurring as large of a computational cost as the O(n 2) complexity of RNs. Sorting-based set functions The use of sorting has often been considered in the set learning literature due to its natural way of ensuring permutation-invariance. The typical approach is to sort elements of the set as units rather than our approach of sorting each feature individually. For example, the similarly-named SortPooling sorts the elements based on one feature of each element. However, this introduces discontinuities into the optimisation whenever two elements swap positions after the sort. For variable-sized sets, they simply truncate (which again adds discontinuities) or pad the sorted list to a fixed length and process this with a CNN, treating the sorted vectors as a sequence. and truncate to a fixed-size set by computing a score for each element and keeping elements with the top-k scores. In contrast, our pooling handles variable set sizes without discontinuities through the featurewise sort and continuous weight space. propose a graph auto-encoder where the decoder use the "inverse" of what the top-k operator does in the encoder, similar to our approach. Instead of numerically sorting, and Zhang et al. (2019b) learn an ordering of set elements instead. Outside of the set learning literature, rank-based pooling in a convolutional neural network has been used in , where the rank is turned into a weight. Sorting within a single feature vector has been used for modeling more powerful functions under a Lipschitz constraint for Wasserstein GANs and improved robustness to adversarial examples . Set prediction Assignment-based losses combined with an MLP or similar are a popular choice for various auto-encoding and generative tasks on point clouds (; ;). An interesting alternative approach is to perform the set generation sequentially (; ; ; . The difficulty lies in how to turn the set into one or multiple sequences, which these papers try to solve in different ways. Since the initial release of this paper, Zhang et al. (2019a) developed a set prediction method which uses FSPool as a core component and motivate their work by our observations about the responsibility problem. Interestingly, their model uses the gradient of the set encoder, which involves computing the gradient of FSPool; this is closely related to the FSUnpool we proposed. We start with two auto-encoder experiments, then move to tasks where we replace the pooling in an established model with FSPool. Full can be found in the appendices, experimental details can be found in Appendix H, and we provide our code for reproducibility at [redacted]. We start with our simple dataset of auto-encoding regular polygons (section 3), with each point in a set corresponding to the x-y coordinate of a vertex in that polygon. This dataset is designed to explicitly test whether the responsibility problem occurs in practice. We keep the set size the same within a training run and only vary the rotation. We try this with set sizes of increasing powers of 2. Model The encoder contains a 2-layer MLP applied to each set element, FSPool, and a 2-layer MLP to produce the latent space. The decoder contains a 2-layer MLP, FSUnpool, and a 2-layer MLP applied on each set element. We train this model to minimise the mean squared error. As baseline, we use a model where the decoder has been replaced with an MLP and train it with either the linear assignment or Chamfer loss (equivalent to AE-EMD and AE-CD models in). Results First, we verified that if the latent space is always zeroed out, the model with FSPool is unable to train, suggesting that the latent space is being used and is necessary. For our training runs with set sizes up to 128, our auto-encoder is able to reconstruct the point set close to perfectly (see Appendix B). Meanwhile, the baseline converges significantly slower with high reconstruction error when the number of points is 8 or fewer and outputs the same set irrespective of input above that, regardless of loss function. Even when significantly increasing the latent size, dimensionality of layers, tweaking the learning rate, and replacing FSPool in the encoder with sum, mean, or max, the baseline trained with the linear assignment or Chamfer loss fails completely at 16 points. We verified that for 4 points, the baseline shows the discontinuous jump behaviour in the outputs as we predict in Figure 1. This experiment highlights the difficulty of learning this simple dataset with traditional approaches due to the responsibility problem, while our model is able to fit this dataset with ease. Next, we turn to the harder task of auto-encoding MNIST images -turned into sets of points -using a denoising auto-encoder. Each pixel that is above the mean pixel level is considered to be part of the set with its x-y coordinates as feature, scaled to be within the range of. The set size varies between examples and is 133 on average. We add Gaussian noise to the points in the set and use the set without noise as training target for the denoising auto-encoder. Model We use exactly the same architecture as on the polygon dataset. As baseline models, we combine sum/mean/max pooling encoders with MLP/LSTM decoders and train with the Chamfer loss. This closely corresponds to the AE-CD approach with the MLP decoder and the model by with the LSTM decoder. We tried the approach by Zhang et al. (2019a), but it performs much worse than the other baselines, likely because it requires a bigger encoder (our encoder has ∼3000 parameters, their encoder has ∼85000 parameters). Results We show example outputs in Figure 3 and the full in Appendix C. We focus on comparing our FSPool-FSUnpool model against the best baseline, which uses the sum pooling encoder and MLP decoder. In general, our model can reconstruct the digits much better than the baseline, which tends to predict too few points even though it always has 342 (the maximum set size) times 2 outputs available. Occasionally, the baseline also makes big errors such as turning 5s into 8s (first σ = 0.01 example), which we have not observed with our model. Instead of auto-encoding MNIST sets, we can also classify them. We use the same dataset and replace the set decoder in our model and the baseline with a 2-layer MLP classifier. We consider three variants: using the trained auto-encoder weights for the encoder and freezing them, not freezing them (finetuning), and training all weights from random initialisation. This tests how informative the learned representations of the pre-trained auto-encoder and the encoder are. Results We show our for σ = 0.05 in Table 1. Results for σ = 0.00 and 100 epochs are shown in Appendix D. Even though our model can store information in the permutation that skips the latent space, our latent space contains more information to correctly classify a set, even when the weights are fixed. Our model with fixed encoder weights already performs better after 1 epoch of training than the baseline models with unfrozen weights after 10 epochs of training. This shows the benefit of the FSPool-FSUnpool auto-encoder to the representation. When allowing the encoder weights to change (Unfrozen and Random init), our again improve significantly over the baselines. Interestingly, switching the relaxed sort to the unrelaxed sort in our model when using the fixed auto-encoder weights does not hurt accuracy. Training the FSPool model takes 45 seconds per epoch on a GTX 1080 GPU, only slightly more than the baselines with 37 seconds per epoch. CLEVR is a visual question answering dataset where the task is to classify an answer to a question about an image. The images show scenes of 3D objects with different attributes, and the task is to answer reasoning questions such as "what size is the sphere that is left of the green thing". Since we are interested in sets, we use this dataset with the ground-truth state description -the set of objects (maximum size 10) and their attributes -as input instead of an image of the rendered scene. Model For this dataset, we compare against relation networks (RN) -explicitly modeling all pairwise relations -Janossy pooling , and regular pooling functions. While the original RN paper reports a of 96.4% for this dataset, we use a tuned implementation by with 2.6% better accuracy. For our model, we modify this to not operate on pairwise relations and replace the existing sum pooling with FSPool. We use the same hyperparameters for our model as the strong RN baseline without further tuning them. Results Over 10 runs, Table 2 shows that our FSPool model reaches the best accuracy and also reaches the listed accuracy milestones in fewer epochs than all baselines. The difference in accuracy is statistically significant (two-tailed t-tests against sum, mean, RN, all with p ≈ 0.01). Also, FSPool reaches 99% accuracy in 5.3 h, while the fastest baseline, mean pooling, reaches the same accuracy in 6.2 h. Surprisingly, RNs do not provide any benefit here, despite the hyperparameters being explicitly tuned for the RN model. We show some of the functions f (·,W) that FSPool has learned in Appendix E. These confirm that FSPool uses more complex functions than just sums or maximums, which allow it to capture more information about the set than other pooling functions. We perform a large number of experiments on various graph classification datasets from the TU repository: 4 graph datasets from bioinformatics (for example with the graph encoding the structure of a molecule) and 5 datasets from social networks (for example with the graph encoding connectivity between people who worked with each other). The task is to classify the whole graph into one of multiple classes such as positive or negative drug response. Model We use the state-of-the-art graph neural network GIN as baseline. This involves a series of graph convolutions (which includes aggregation of features from each node's set of neighbours into the node), a readout (which aggregates the set of all nodes into one feature vector), and a classification with an MLP. We replace the usual sum or mean pooling readout with FSPool k = 5 for our model. We repeat 10-fold cross-validation on each dataset 10 times and use the same hyperparameter ranges as for our model and the GIN baseline. Results We show the in Appendix F. On 6 out of 9 datasets, FSPool achieves better test accuracy. On a different 6 datasets, it converges to the best validation accuracy faster. A Wilcoxon signed-rank test shows that the difference in accuracy to the standard GIN has p ≈ 0.07 (W = 7) and the difference in convergence speed has p ≈ 0.11 (W = 9). Keep in mind that just because the have p > 0.05, it does not mean that the are invalid. Zhang et al. (2019a) build on the ideas in this paper to develop a model that can predict sets from an image. Their model requires from a set encoder that the more similar two set inputs are, the more similar their representations should be. This is harder than classification, because different inputs (of the same class) should no longer map to the same representation. In this experiment, we quantify the benefit of the RN + FSPool set encoder they used. We use their experimental set-up and replace FSPool with sum (this gives the normal RN model) or max pooling. We train this on CLEVR to predict the set of bounding boxes or the state description (this was the input in subsection 6.3). Results Appendix G shows that for both bounding box and state prediction models, the RN encoder using FSPool is much better than sum or max. This shows that it is possible to improve on standard Relation Networks simply by replacing the sum with FSPool when the task is challenging enough. In this paper, we identified the responsibility problem with existing approaches for predicting sets and introduced FSPool, which provides a way around this issue in auto-encoders. In experiments on two datasets of point clouds, we showed that this in much better reconstructions. We believe that this is an important step towards set prediction tasks with more complex set elements. However, because our decoder uses information from the encoder, it is not easily possible to turn it into a generative set model, which is the main limitation of our approach. Still, we find that using the auto-encoder to obtain better representations and pre-trained weights can be beneficial by itself. Our insights about the responsibility problem have already been successfully used to create a model without the limitations of our auto-encoder (a). In classification experiments, we also showed that simply replacing the pooling function in an existing model with FSPool can give us better and faster convergence. We showed that FSPool consistently learns better set representations at a relatively small computational cost, leading to improved in the downstream task. Our model thus has immediate applications in various types of set models that have traditionally used sum or max pooling. It would be useful to theoretically characterise what types of relations are more easily expressed by FSPool through an analysis like in. This may in further insights into how to learn better set representations efficiently. The following theorem is a more formal treatment of the responsibility problem ing in discontinuities. Theorem 1. For any set function f: n is the set of all sets of size n with elements in R d ) from a set of points S = {x 1, x 2, . . ., x n} to a list representation of that set L = [x σ, x σ,..., x σ(n) ] with some fixed permutation σ ∈ Π, there will be a discontinuity in f: there exists an ε > 0 such that for all δ > 0, there exist two sets S 1 and S 2 where: d s is a measure of the distance between two sets (e.g. Chamfer loss) and d l is the sum of Euclidean θ Figure 4: Example of the set with two points. Proof. We prove the theorem by considering mappings from a set of two points in two dimensions. For larger sets or sets with more dimensions, we can isolate two points and two dimensions and ignore the remaining points and dimensions. Let us consider the set of two points S(θ) = while for θ > θ *, the list representation will Let ε = 3.9 and δ be given. We can find a sufficiently small α > 0 so that The reason why this does not apply to our method is that rather than choosing a fixed σ for the list representation, the permutation-equivariance (instead of the invariance of set functions) allows our model to have L(π) = L. Results In Table 3, Table 4, and Table 5, we show the of various model and training loss combinations. We include a random baseline that outputs a polygon with the correct size and centre, but random rotation. These show that FSPool with the direct MSE training loss is clearly better than the baseline with either linear assignment or Chamfer loss on all the evaluation metrics. When the set size is 16 or greater, the other combinations only perform as well as the random baseline because they output the same constant set regardless of input. Results We show the for the default MNIST setting in Table 6. Interestingly, the sum pooling baseline has a lower Chamfer reconstruction error than our model, despite the example outputs in Figure 3 looking clearly worse. This demonstrates a weakness of the Chamfer loss. Our model avoids this weakness by being trained with a normal MSE loss (with the cost of a potentially higher Chamfer loss), which is not possible with the baselines. The sum pooling baseline has a better test Chamfer loss because it is trained to minimise it, but it is also solving an easier task, since it does not need to distinguish padding from non-padding elements. The main reason for this difference comes from the shortcoming of the Chamfer loss in distinguishing sets with duplicates or near-duplicates. For example, the Chamfer loss between [1, 1.001, 9] and [1, 9, 9.001] is close to 0. Most points in an MNIST set are quite close to many other points and there are many duplicate padding elements, so this problem with the Chamfer loss is certainly present on MNIST. That is why minimising MSE can lead to different with higher Chamfer loss than minimising Chamfer loss directly, even though the qualitative seem worse for the latter. We can make the comparison between our model and the baselines more similar by forcing the models to predict an additional "mask feature" for each set element. This takes the value 1 when the point is present (non-padding element) and 0 (padding element) when not. This setting is useful for tasks where the predicted set size matters, as it allows points at the coordinates to be distinguished from padding elements. These padding elements are necessary for efficient minibatch-wise training. The of this variant are shown in Table 7. Now, our model is clearly better: even though our auto-encoder minimises an MSE loss, the test Chamfer loss is also much better than all the baselines. Having to predict this additional mask feature does not affect our model predictions much because our model structure lets our model "know" which elements are padding elements, while this is much more challenging for the baselines. Table 9: Classification accuracy (mean ± stdev) on MNIST for 100 epochs over 6 runs. Results Table 8 and Table 9 show the for σ = 0.00 and for 100 epochs for both σ = 0.05 and σ = 0.00 respectively. Note that these are based on pre-trained models from the default MNIST setting without mask feature. Like before, the FSPool-based models are consistently superior to all the baselines. Note that while report an accuracy of ∼99% on a similar set version of MNIST, our model uses noisy sets as input and is much smaller and simpler: we have 3820 parameters, while their model has 1.6 million parameters. Our model also does not use dropout, batch norm, a branching network architecture, and a stepped learning rate schedule. When we try to match their model size, our accuracies for σ = 0.00 increase to ∼99% as well. E CLEVR Figure 5: Shapes of piecewise linear functions learned by the FSPool model on CLEVR. These show r ∈ on the x-axis and f (r,w) on the y-axis for a particularw of a fully-trained model. A common shape among these functions are variants of max pooling: close to 0 weight for most ranks and a large non-zero weight on either the maximum or the minimum value, for example in row 2 column 2. There are many functions that simple maximums or sums can not easily represent, such as a variant of max pooling with the values slightly below the max receiving a weight of the opposite sign (see row 1 column 1) or the shape in the penultimate row column 5. The functions shown here may have a stronger tendency towards 0 values than normal due to the use of weight decay on CLEVR. Experimental setup The datasets and node features used are the same as in GIN; we did not cherry-pick them. Because the social network datasets are purely structural without node features, a 70.0 ±0.9 47.8 ±0.9 --73.8 ±0.5 DIFFPOOL ----75.5 WL* 73.8 50.9 81.0 52.5 78.9 GIN-BASE* 75 67.0 61.3 56.6 62.6 PATCHY-SAN 92.6 ±4.2 75.9 ±2.8 60.0 ±4.8 78.6 ±1.9 SORTPOOL 85.8 ±1.7 75.5 ±0.9 58.6 ±2.5 74.4 ±0.5 DIFFPOOL -76.3 --WL 84.1 ±1.9 74.7 ±0.5 58.0 ±2.5 85.5 ±0.5 WL* 90.4 75.0 59.9 86.0 GIN-BASE* 89 constant 1 feature is used on the RDT datasets and the one-hot-encoded node degree is used on the other social network datasets. The hyperparameter sweep is done based on best validation accuracy for each fold in the cross-validation individually and over the same combinations as specified in GIN. Note that in GIN, hyperparameters are selected based on best test accuracy. This is a problem, because they consider the number of epochs a hyperparameter when accuracies tend to significantly vary between individual epochs. For example, our average on the PROTEINS dataset would change from 73.8% to 77.1% if we were to select based on best test accuracy, which would be better than their 76.2%. While we initially also used k = 20 in FSPool for this experiment, we found that k = 5 was consistently an improvement. The k = 20 model was still better than the baseline on average by a smaller margin. Results We show our of GIN-FSPool and the GIN baseline averaged over 10 repeats in Table 10. On the majority of datasets, FSPool has slightly better accuracies than the strong baseline and consistently takes fewer epochs to reach its highest validation accuracy. On the two RDT datasets, this improvement is large. Interestingly, these are the two datasets where the number of nodes to be pooled is by far the largest with an average of 400+ nodes per graph, compared to the next largest COLLAB with an average of 75 nodes. This is perhaps evidence that FSPool is helping to avoid the bottleneck problem of pooling a large set of feature vectors to a single feature vector. We emphasise that the main comparison to be made is between the GIN-Base and the GIN-FSPool model, since that is the only comparison where the only factor of difference is the pooling method. When comparing against other models, the network architecture, training hyperparameters, and evaluation methodology can differ significantly. Keep in mind that while GIN-Base looks much worse than the original GIN-Base*, the difference is that our implementation has hyperparameters properly selected by validation accuracy, while GINBase* selected them by test accuracy. If we were to select based on test accuracy, our implementation frequently outperforms their . Also, they only performed a single run of 10-fold crossvalidation. G DEEP SET PREDICTION NETWORKS Results Table 11 and Table 12 show that the FSPool-based RN encoder is much better than any of the baselines. The representation of DSPN-RN-FSPool is good enough that iterating the DSPN algorithm for more steps than the model was trained with can benefit the prediction, while for the baselines it generally just worsens. This is especially apparent for the harder dataset of state prediction, where more information has to be compressed into the latent space. We provide the code to reproduce all experiments at [redacted]. For almost all experiments, we used FSPool and the unpooling version of it with k = 20. We guessed this value without tuning, and we did not observe any major differences when we tried to change this on CLEVR to k = 5 and k = 40.W can be initialised in different ways, such as by sampling from a standard Gaussian. However, for the purposes of starting the model as similarly as possible to the sum pooling baseline on CLEVR and on the graph classification datasets, we initialiseW to a matrix of all 1s on them. The polygons are centred on 0 with a radius of 1. The points in the set are randomly permuted to remove any ordering in the set from the generation process that a model that is not permutationinvariant or permutation-equivariant could exploit. We use a batch size of 16 for all three models and train it for 10240 steps. We use the Adam optimiser with 0.001 learning rate and their suggested values for the other optimiser parameters (PyTorch defaults). Weights of linear and convolutional layers are initialised as suggested in. The size of every hidden layer is set to 16 and the latent space is set to 1 (it should only need to store the rotation as latent variable). We have also tried much hidden and latent space sizes of 128 when we tried to get better for the baselines. We train on the training set of MNIST for 10 epochs and the shown come from the test set of MNIST. For an image, the coordinate of a pixel is included if the pixel is above the mean pixel level of 0.1307 (with pixel levels ranging 0-1). Again, the order of the points are randomised. We did not include of the linear assignment loss because we did not get the model to converge to of similar quality to the direct MSE loss or Chamfer loss, and training time took too long (> 1 day) in order to find better parameters. The latent space is increased from 1 to 16 and the size of the hidden layers is increased from 16 to 32. All other hyperparameters are the the same as for the Polygons dataset. The architecture and hyperparameters come from the third-party open-source implementation available at https://github.com/mesnico/RelationNetworks-CLEVR. For the RN baseline, the set is first expanded into the set of all pairs by concatenating the 2 feature vectors of the pair for all pairs of elements in the set. For the Janossy Pooling baseline, we use the model configuration from that appeared best in their experiments, which uses π-SGD with an LSTM that has |h| as neighbourhood size. The question representation coming from the 256-unit LSTM, processing the question tokens in reverse with each token embedded into 32 dimensions, is concatenated to all elements in the set. Each element of this new set is first processed by a 4-layer MLP with 512 neurons in each layer and ReLU activations. The set of feature vectors is pooled with a pooling method like sum and the output of this is processed with a 3-layer MLP (hidden sizes 512, 1024, and number of answer classes) with ReLU activations. A dropout rate of 0.05 is applied before the last layer of this MLP. Adam is used with a starting learning rate of 0.000005, which doubles every 20 epochs until the maximum learning rate of 0.0005 is reached. Weight decay of 0.0001 is applied. The model is trained for 350 epochs. The GIN architecture starts with 5 sequential blocks of graph convolutions. Each block starts with summing the feature vector of each node's neighbours into the node's own feature vector. Then, an MLP is applied to the feature vectors of all the nodes individually. The details of this MLP were somewhat unclear in and we chose Linear-ReLU-BN-Linear-ReLU-BN in the end. We tried Linear-BN-ReLU-Linear-BN-ReLU as well, which gave us slightly worse validation for both the baseline and the FSPool version. The outputs of each of the 5 blocks are concatenated and pooled, either with a sum for the social network datasets, mean for the social network datasets (this is as specified in GIN), or with FSPool for both types of datasets. This is followed by BNLinear-ReLU-Dropout-Linear as classifier with a softmax output and cross-entropy loss. We used the torch-geometric library to implement this model. The starting learning rate for Adam is 0.01 and is reduced every 50 epochs. Weights are initialised as suggested in. The hyperparameters to choose from are: dropout ratio ∈ {0, 0.5}, batch size ∈ {32, 128}, if bioinformatics dataset hidden sizes of all layers ∈ {16, 32} and 500 epochs, if social network dataset the hidden size is 64 and 250 epochs. Due to GPU memory limitations we used a batch size of 100 instead of 128 for social network datasets. The best hyperparameters are selected based on best average validation accuracy across the 10-fold cross-validation, where one of the 9 training folds is used as validation set each time. In other words, within one 10-fold cross-validation run the hyperparameters used for the test set are the same, while across the 10 repeats of this with different seeds the best hyperparameters may differ. The architecture and hyperparameters come from the third-party open-source implementation available at https://github.com/Cyanogenoid/dspn. The only thing we change from this is replacing the pooling in the RN. All other hyperparameters are kept the same. The input image is encoded with a ResNet-34 with two additional convolutional layers with 512 filters and stride two to obtain a feature vector for the image. This feature vector is decoded into a set using the DSPN algorithm, which requires encoding an intermediate set with the set encoder and performing gradient descent on it. This set encoder creates all pairs of sets like in normal RNs, processes each pair with a 2-layer MLP with 512 neurons with one ReLU activation in the middle, then pools this into a feature vector. The intermediate set is updated with the gradient 10 times in training, but can be iterated a different amount in evaluation. The model is trained to minimise the linear assignment loss with the Adam optimiser for 100 epochs using a learning rate of 0.0003.

Represent the short summary for finding the corresponding scientific article (long)

Sort in encoder and undo sorting in decoder to avoid responsibility problem in set auto-encoders

represent this text to find a much much longer article about it with say hundreds-thousands of words

Information about whether a neural network's output will be correct or incorrect is somewhat present in the outputs of the network's intermediate layers.