Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Audio signals are sampled at high temporal resolutions, and learning to synthesize audio requires capturing structure across a range of timescales. Generative adversarial networks (GANs) have seen wide success at generating images that are both locally and globally coherent, but they have seen little application to audio generation. In this paper we introduce WaveGAN, a first attempt at applying GANs to unsupervised synthesis of raw-waveform audio. WaveGAN is capable of synthesizing one second slices of audio waveforms with global coherence, suitable for sound effect generation. Our experiments demonstrate that—without labels—WaveGAN learns to produce intelligible words when trained on a small-vocabulary speech dataset, and can also synthesize audio from other domains such as drums, bird vocalizations, and piano. We compare WaveGAN to a method which applies GANs designed for image generation on image-like audio feature representations, finding both approaches to be promising. Synthesizing audio for specific domains has many practical applications in creative sound design for music and film. Musicians and Foley artists scour large databases of sound effects to find particular audio recordings suitable for specific scenarios. This strategy is painstaking and may in a negative outcome if the ideal sound effect does not exist in the library. A better approach might allow a sound artist to explore a compact latent space of audio, taking broad steps to find the types of sounds they are looking for (e.g. footsteps) and making small adjustments to latent variables to finetune (e.g. a large boot lands on a gravel path). However, audio signals have high temporal resolution, and strategies that learn such a representation must perform effectively in high dimensions. Generative Adversarial Networks (GANs) are one such unsupervised strategy for mapping low-dimensional latent vectors to high-dimensional data. The potential advantages of GAN-based approaches to audio synthesis are numerous. Firstly, GANs could be useful for data augmentation in data-hungry speech recognition systems. Secondly, GANs could enable rapid and straightforward sampling of large amounts of audio. Furthermore, while the usefulness of generating static images with GANs is arguable, there are many applications (e.g. Foley) for which generating sound effects is immediately useful. But despite their increasing fidelity at synthesizing images (; BID2), GANs have yet to be demonstrated capable of synthesizing audio in an unsupervised setting. A naïve solution for applying image-generating GANs to audio would be to operate them on imagelike spectrograms, i.e., time-frequency representations of audio. This practice of bootstrapping image recognition algorithms for audio tasks is commonplace in the discriminative setting . In the generative setting however, this approach is problematic as the most perceptually-informed spectrograms are non-invertible, and hence cannot be listened to without lossy estimations or learned inversion models .Recent work (van den ;) has shown that neural networks can be trained with autoregression to operate on raw audio. Such approaches are attractive as they dispense with engineered feature representations. However, unlike with GANs, the autoregressive setting in slow generation as output audio samples must be fed back into the model one at a time. In this work, we investigate both waveform and spectrogram strategies for generating one-second slices of audio with GANs.1 For our spectrogram approach (SpecGAN), we first design a spectrogram representation that allows for approximate inversion, and bootstrap the two-dimensional deep convolutional GAN (DCGAN) method to operate on these spectrograms. In WaveGAN, our waveform approach, we flatten the DCGAN architecture to operate in one dimension, ing in a model with the same number of parameters and numerical operations as its twodimensional analog. With WaveGAN, we provide both a starting point for practical audio synthesis with GANs and a recipe for modifying other image generation methods to operate on waveforms. We primarily envisage our method being applied to the generation of short sound effects suitable for use in music and film. For example, we trained a WaveGAN on drums, ing in a procedural drum machine designed to assist electronic musicians (demo chrisdonahue.com/wavegan). However, human evaluation for such domain-specific tasks would require expert listeners. Therefore, we also consider a speech benchmark, facilitating straightforward assessment by human annotators. Specifically, we explore a task where success can easily be judged by any English speaker: generating examples of spoken digits "zero" through "nine".Though our evaluation focuses on a speech generation task, we note that it is not our goal to develop a text-to-speech synthesizer. Instead, our investigation concerns whether unsupervised strategies can learn global structure (e.g. words in speech data) implicit in high-dimensional audio signals without conditioning. Our experiments on speech demonstrate that both WaveGAN and SpecGAN can generate spoken digits that are intelligible to humans. On criteria of sound quality and speaker diversity, human judges indicate a preference for the audio generated by WaveGAN compared to that from SpecGAN. GANs learn mappings from low-dimensional latent vectors z ∈ Z, i.i.d. samples from known prior P Z, to points in the space of natural data X. In their original formulation , a generator G: Z → X is pitted against a discriminator D: X → in a two-player minimax game. G is trained to minimize the following value function, while D is trained to maximize it: DISPLAYFORM0 In other words, D is trained to determine if an example is real or fake, and G is trained to fool the discriminator into thinking its output is real. demonstrate that their proposed training algorithm for Equation 1 equates to minimizing the Jensen-Shannon divergence between P X, the data distribution, and P G, the implicit distribution of the generator when z ∼ P Z. In this original formulation, GANs are notoriously difficult to train, and prone to catastrophic failure cases. Instead of Jensen-Shannon divergence, BID1 suggest minimizing the smoother Wasserstein-1 distance between generated and data distributions DISPLAYFORM1 where f L ≤ 1: X → R is the family of functions that are 1-Lipschitz. To minimize Wasserstein distance, they suggest a GAN training algorithm (WGAN), similar to that of , for the following value function: DISPLAYFORM2 Figure 1: First eight principal components for 5x5 patches from natural images (left) versus those of length-25 audio slices from speech (right). Periodic patterns are unusual in natural images but a fundamental structure in audio. Figure 2: Depiction of the transposed convolution operation for the first layers of the DCGAN (left) and WaveGAN (right) generators. DCGAN uses small (5x5), twodimensional filters while WaveGAN uses longer (length-25), one-dimensional filters and a larger upsampling factor. Both strategies have the same number of parameters and numerical operations. We motivate our design choices for WaveGAN by first highlighting the different types of structure found in audio versus images. One way to illustrate the differences between audio and images is by examining the axes along which these types of data vary most substantially, i.e. by principal component analysis. In Figure 1, we show the first eight principal components for patches from natural images and slices from speech. While the principal components of images generally capture intensity, gradient, and edge characteristics, those from audio form a periodic basis that decompose the audio into constituent frequency bands. In general, natural audio signals are more likely to exhibit periodicity than natural images. As a consequence, correlations across large windows are commonplace in audio. For example, in a waveform sampled at 16 kHz, a 440 Hz sinusoid (the musical note A4) takes over 36 samples to complete a single cycle. This suggests that filters with larger receptive fields are needed to process raw audio. This same intuition motivated van den in their design of WaveNet, which uses dilated convolutions to exponentially increase the model's effective receptive field with linear increase in layer depth. We base our WaveGAN architecture off of DCGAN which popularized usage of GANs for image synthesis. The DCGAN generator uses the transposed convolution operation (Figure 2) to iteratively upsample low-resolution feature maps into a high-resolution image. Motivated by our above discussion, we modify this transposed convolution operation to widen its receptive field. Specifically, we use longer one-dimensional filters of length 25 instead of two-dimensional filters of size 5x5, and we upsample by a factor of 4 instead of 2 at each layer (Figure 2). We modify the discriminator in a similar way, using length-25 filters in one dimension and increasing stride from 2 to 4. These changes in WaveGAN having the same number of parameters, numerical operations, and output dimensionality as DCGAN.Because DCGAN outputs 64x64 pixel images -equivalent to just 4096 audio samples -we add one additional layer to the model ing in 16384 samples, slightly more than one second of audio at 16 kHz. This length is already sufficient for certain sound domains (e.g. sound effects, voice commands), and future work adapting megapixel image generation techniques could expand the output length to more than a minute. We requantize the real data from its 16-bit integer representation (linear pulse code modulation) to 32-bit floating point, and our generator similarly outputs floating point waveforms. A complete description of our model is in Appendix D.In summary, we outline our modifications to the DCGAN method which in WaveGAN. This straightforward recipe already produces reasonable audio, and further contributions outlined below and in Appendix A serve to refine .1. Flatten 2D convolutions into 1D (e.g. 5x5 2D convolution becomes length-25 1D).2. Increase the stride factor for all convolutions (e.g. stride 2x2 becomes stride 4).3. Remove batch normalization from the generator and discriminator.4. Train using the WGAN-GP strategy. Phase shuffle n=1 DISPLAYFORM0 Figure 3: At each layer of the Wave-GAN discriminator, the phase shuffle operation perturbs the phase of each feature map by Uniform ∼ [−n, n] samples, filling in the missing samples (dashed outlines) by reflection. Here we depict all possible outcomes for a layer with four feature maps (n = 1).Generative image models that upsample by transposed convolution (such as DCGAN) are known to produce characteristic "checkerboard" artifacts in images . Periodic patterns are less common in images (Section 3.1), and thus the discriminator can learn to reject images that contain them. For audio, analogous artifacts are perceived as pitched noise which may overlap with frequencies commonplace in the real data, making the discriminator's objective more challenging. However, the artifact frequencies will always occur at a particular phase, allowing the discriminator to learn a trivial policy to reject generated examples. This may inhibit the overall optimization problem. To prevent the discriminator from learning such a solution, we propose the phase shuffle operation with hyperparameter n. Phase shuffle randomly perturbs the phase of each layer's activations by −n to n samples before input to the next layer (Figure 3). We apply phase shuffle only to the discriminator, as the latent vector already provides the generator a mechanism to manipulate the phase of a ant waveform. Intuitively speaking, phase shuffle makes the discriminator's job more challenging by requiring invariance to the phase of the input waveform. While a minority of recent research in discriminative audio classification tasks has used raw audio input , most of these approaches operate on spectrogram representations of audio. A generative model may also benefit from operating in such a time-frequency space. However, commonly-used representations in the discriminative setting are uninvertible. With SpecGAN, our frequency-domain audio generation model, we design a spectrogram representation that is both well-suited to GANs designed for image generation and can be approximately inverted. Additionally, to facilitate direct comparison, our representation is designed to use the same dimensionality per unit of time as WaveGAN (16384 samples yield a 128x128 spectrogram).To process audio into suitable spectrograms, we first perform the short-time Fourier transform with 16 ms windows and 8 ms stride, ing in 128 frequency bins 2 linearly spaced from 0 to 8 kHz. We take the magnitude of the ant spectra and scale amplitude values logarithmically to better-align with human perception. We then normalize each frequency bin to have zero mean and unit variance. This type of preprocessing is commonplace in audio classification, but produce spectrograms with unbounded values-a departure from image representations. We therefore clip the spectra to 3 standard deviations and rescale to [−1, 1]. Through an informal listening test, we determined that this clipping strategy did not produce an audible difference during inversion. Once our dataset has been processed into this format, we operate the DCGAN algorithm on the ant spectra. To render the ant generated spectrograms as waveforms, we first invert the steps of spectrogram preprocessing described above, ing in linear-amplitude magnitude spectra. We then employ the iterative Griffin-Lim algorithm with 16 iterations to estimate phase and produce 16384 audio samples. To facilitate human evaluation, our experimentation focuses on the Speech Commands Dataset . This dataset consists of many speakers recording individual words in uncontrolled recording conditions. We explore a subset consisting of the spoken digits "zero" through "nine" and refer to this subset as the Speech Commands Zero Through Nine (SC09) dataset. While this dataset is intentionally reminiscent of the popular MNIST dataset of written digits, we note that examples from SC09 are much higher dimensional (R 16000) than examples from MNIST (R 28×28=784).These ten words encompass many phonemes and two consist of multiple syllables. Each recording is one second in length, and we do not attempt to align the words in time. There are 1850 utterances of each word in the training set, ing in 5.3 hours of speech. The heterogeneity of alignments, speakers, and recording conditions make this a challenging dataset for generative modeling. Our baseline configuration for WaveGAN excludes phase shuffle. We compare this to the performance of WaveGAN with phase shuffle (n ∈ {2, 4}) and a variant of WaveGAN which uses nearest-neighbor upsampling rather than transposed convolution . Hoping to reduce noisy artifacts, we also experiment with adding a wide (length-512) post-processing filter to the output of the generator and learning its parameters with the rest of the generator variables (details in Appendix A.1). We use the WGAN-GP algorithm for all experiments, find-ing it to produce reasonable where others (; ; BID1 failed. We compare the performance of these configurations to that of SpecGAN.We also perform experiments on four other datasets with different characteristics FIG0):1. Drum sound effects (0.7 hours): Drum samples for kicks, snares, toms, and cymbals 2. Bird vocalizations (12.2 hours): In-the-wild recordings of many species 3. Piano (0.3 hours): Professional performer playing a variety of Bach compositions 4. Large vocab speech (TIMIT) (2.4 hours): Multiple speakers, clean BID12 We train our networks using batches of size 64 on a single NVIDIA P100 GPU. During our quantitative evaluation of SC09 (discussed below), our WaveGAN networks converge by their early stopping criteria (inception score) within four days (200k iterations, around 3500 epochs), and produce speech-like audio within the first hour of training. Our SpecGAN networks converge more quickly, within two days (around 1750 epochs). On the other four datasets, we train WaveGAN for 200k iterations representing nearly 1500 epochs for the largest dataset. Unlike with autoregressive methods (van den ;), generation with WaveGAN is fully parallel and can produce an hour of audio in less than two seconds. We list all hyperparameters in Appendix E. Evaluation of generative models is a fraught topic. demonstrate that quantitative measures of sample quality are poorly correlated with each other and human judgement. Accordingly, we use several quantitative evaluation metrics for hyperparameter validation and discussion, and also evaluate our most promising models with human judges.6.1 propose the inception score, which uses a pre-trained Inception classifier to measure both the diversity and semantic discriminability of generated images, finding that the measure correlates well with human judgement. Given model scores P (y | x) with marginal P (y), the inception score is defined as exp(E x D KL (P (y | x)||P (y))), and is estimated over a large number of samples (e.g. 50k). For n classes, this measure ranges from 1 to n, and is maximized when the model is completely confident about each prediction and predicts each label equally often. We will use this measure as our primary quantitative evaluation method and early stopping criteria. To measure inception score, we train an audio classifier on SC09. Our classifier first computes a short-time Fourier transform of the input audio with 64 ms windows and 8 ms stride. This representation is projected to 128 frequency bins equally spaced on the Mel scale from 40 Hz to 7800 Hz. Amplitudes are scaled logarithmically and normalized so that each bin has zero mean and unit variance. We process this perceptually-informed representation with four layers of convolution and pooling, projecting the to a softmax layer with 10 classes. We perform early stopping on the minimum negative log-likelihood of the validation set; the ant model achieves 93% accuracy on the test set. Because this classifier observes spectrograms, our spectrogramgenerating models may have a representational advantage over our waveform-generating models. Inception score has two trivial failure cases in which a poor generative model can achieve a high score. Firstly, a generative model that outputs a single example of each class with uniform probability will be assigned a high score. Secondly, a generative model that overfits the training data will achieve a high score simply by outputting examples on which the classifier was trained. We use two indicators metrics to determine if a high inception score has been caused by either of these two undesirable cases. Our first indicator, |D| self, measures the average Euclidean distance of a set of 1k examples to their nearest neighbor within the set (other than itself). A higher |D| self indicates higher diversity amongst samples. Because measuring Euclidean distance in time-domain audio poorly represents human perception, we evaluate distances in the same frequency-domain representation as our classifier from Section 6.1.Our second indicator, |D| train, measures the average Euclidean distance of 1k examples to their nearest neighbor in the real training data. If the generative model simply produces examples from the training set, this measure will be 0. We report |D| train and |D| self relative to those of the test set. While inception score is a useful metric for hyperparameter validation, our ultimate goal is to produce examples that are intelligible to humans. To this end, we measure the ability of human annotators on Amazon Mechanical Turk to label the generated audio. Using our best WaveGAN and SpecGAN models as measured by inception score, we generate random examples until we have 300 for each digit (as labeled by our classifier from Section 6.1)-3000 total. In batches of ten random examples, we ask annotators to label which digit they perceive in each example, and compute their accuracy with respect to the classifier's labels (random accuracy would be 10%). After each batch, annotators assign subjective values of 1-5 for criteria of sound quality, ease of intelligibility, and speaker diversity. We report accuracy (n = 3000) and mean opinion scores (n = 300) in TAB0. Results for our evaluation appear in TAB0. We also evaluate our metrics on the real training data, the real test data, and a version of SC09 generated by a parametric speech synthesizer BID4. We also compare to SampleRNN and two public implementations of WaveNet (van den), but neither method produced competitive (details in Appendix B), and we excluded them from further evaluation. These autoregressive models have not previously been examined on small-vocabulary speech data, and their success at generating full words has only been demonstrated when conditioning on rich linguistic features. Sound examples for all experiments can be found at chrisdonahue.com/wavegan_examples.While the maximum inception score for SC09 is 10, any score higher than the test set score of 8 should be seen as evidence that a generative model has overfit. Our best WaveGAN model uses phase shuffle with n = 2 and achieves an inception score of 4.7. To compare the effect of phase shuffle to other common regularizers, we also tried using 50% dropout in the discriminator's activations, which ed in a lower score. Phase shuffle decreased the inception score of SpecGAN, possibly because the operation has an exaggerated effect when applied to the compact temporal axis of spectrograms. Most experiments produced |D| self (diversity) values higher than that of the test data, and all experiments produced |D| train (distance from training data) values higher than that of the test data. While these measures indicate that our generative models produce examples with statistics that deviate from those of the real data, neither metric indicates that the models achieve high inception scores by the trivial solutions outlined in Section 6.2.Compared to examples from WaveGAN, examples from SpecGAN achieve higher inception score (6.0 vs. 4.7) and are labeled more accurately by humans (66% vs. 58%). However, on subjective criteria of sound quality and speaker diversity, humans indicate a preference for examples from WaveGAN. It appears that SpecGAN might better capture the variance in the underlying data compared to WaveGAN, but its success is compromised by sound quality issues when its spectrograms are inverted to audio. It is possible that the poor qualitative ratings for examples from SpecGAN are primarily caused by the lossy Griffin-Lim inversion and not the generative procedure itself. We see promise in both waveform and spectrogram audio generation with GANs; our study does not suggest a decisive winner. For a more thorough investigation of spectrogram generation methods, we point to follow-up work BID10.Finally, we train WaveGAN and SpecGAN models on the four other domains listed in Section 5. Somewhat surprisingly, we find that the frequency-domain spectra produced by WaveGAN (a timedomain method) are visually more consistent with the training data (e.g. in terms of sharpness) than those produced by SpecGAN FIG0 Much of the work within generative modeling of audio is within the context of text-to-speech. Textto-speech systems are primarily either concatenative or parametric. In concatenative systems, audio is generated by sequencing small, prerecorded portions of speech from a phonetically-indexed dictionary . Parametric systems map text to salient parameters of speech, which are then synthesized by a vocoder BID8; see for a comprehensive review. Some of these systems use learning-based approaches such as a hidden Markov models , and separately-trained neural networks pipelines to estimate speech parameters. Recently, several researchers have investigated parametric speech synthesis with end-to-end neural network approaches that learn to produce vocoder features directly from text or phonetic embeddings BID0;;; ). These vocoder features are synthesized to raw audio using off-the-shelf methods such as WORLD and Griffin-Lim , or trained neural vocoders (; ;). All of these methods are supervised: they are trained to map linguistic features to audio outputs. Several approaches have explored unsupervised generation of raw audio. van den propose WaveNet, a convolutional model which learns to predict raw audio samples by autoregressive modeling. WaveNets conditioned on rich linguistic features have widely been deployed in textto-speech systems, though they have not been demonstrated capable of generating cohesive words in the unconditional setting. BID9 BID7 all use GANs in combination with unstructured losses to map spectrograms in one domain to spectrograms in another. BID5 use GANs to map musical performance images into spectrograms. We present WaveGAN, the first application of GANs to unsupervised audio generation. WaveGAN is fully parallelizable and can generate hours of audio in only a few seconds. In its current form, WaveGAN can be used for creative sound design in multimedia production. In our future work we plan to extend WaveGAN to operate on variable-length audio and also explore a variety of label conditioning strategies. By providing a template for modifying image generation models to operate on audio, we hope that this work catalyzes future investigation of GANs for audio synthesis. Post-processing filters reject frequencies corresponding to noise byproducts created by the generative procedure (top). The filter for speech boosts signal in prominent speech bands, while the filter for bird vocalizations (which are more uniformly-distributed in frequency) simply reduces noise presence. Generative models that upsample by transposed convolution are known to produce characteristic "checkerboard" artifacts in images , artifacts with particular spatial periodicities. The discriminator of image-generating GANs can learn to reject images with these artifacts because they are uncommon in real data (as discussed in Section 3.1). However, in the audio domain, the discriminator might not have such luxury as these artifacts correspond to frequencies which might rightfully appear in the real data. While checkerboard artifacts are an annoyance in image generation, they can be devastating to audio generation . While our eye may perceive these types of periodic distortions as an intrusive texture, our ear perceives them as an abrasive tone. To characterize these artifacts in WaveGAN, we measure its impulse response by randomly initializing it 1000 times and passing unit impulses to its first convolutional layer. In FIG2, we plot the average of these responses in the frequency domain. The response has sharp peaks at linear multiples of the sample rates of each convolutional layer (250 Hz, 1 kHz, 4 kHz, etc.). This is in agreement with our informal observation of from WaveGAN, which often have a pitched noise close to the musical note B (247 × 2 n Hz).Below, we will discuss strategies we designed to mitigate these artifacts in WaveGAN. We experiment with adding a post-processing filter to the generator, giving WaveGAN a simple mechanism to filter out undesirable frequencies created by the generative process. This filter has a long window (512 samples) allowing it to represent intricate transfer functions, and the weights of the filter are learned as part of the generator's parameters. In FIG2, we compare the postprocessing filters that WaveGAN learns for human speech and bird vocalizations. The filters boost signal in regions of the frequency spectrum that are most prominent in the real data domain, and introduce notches at bands that are artifacts of the generative procedure as discussed in the previous section. Transposed convolution upsamples signals by inserting zeros in between samples and applying a learned filterbank. This operation introduces aliased frequencies, copies of pre-existing frequencies shifted by multiples of the new Nyquist rate, into the upsampled signal. While aliased frequencies are usually seen as undesirable artifacts of a bad upsampling procedure, in the generative setting their existence may be crucial for producing fine-grained details in the output. We experiment with three other upsampling strategies in WaveGAN: nearest-neighbor, linear and cubic interpolation, all of which attenuate aliased frequencies. In FIG3, we compare these strategies visually. While nearest neighbor upsampling ed in similar audio output to transposed convolution, linear and cubic interpolation strategies ed in qualitatively poor audio output (sound examples: chrisdonahue.com/wavegan_examples). We hypothesize that the aliased frequencies produced by upsampling convolutions may be more critical to audio generation than image generation. We developed our WaveGAN and SpecGAN models primarily to address the task of steerable sound effect generation. This is an inherently different task than text to speech (TTS), however autoregressive waveform models (e.g. WaveNet (van den) and SampleRNN ) that were developed for TTS can also be used to model and generate waveforms unconditionally. Hence, a comparison to these models for our task is reasonable. One upside of autoregressive models for our task is that they have the potential to produce high-quality audio. Potential downsides are 1) these models take several orders of magnitude longer to generate waveforms, and 2) they do not learn a compact latent space of waveforms, causing useful sound generation tasks like continuous exploration and interpolation to be impossible. We attempt to train two public implementations of WaveNet (ImplA 3 and ImplB 4) and SampleRNN failed to produce cohesive words (you can judge for yourself from our sound examples at the bottom chrisdonahue.com/wavegan_examples). This poor subjective performance is echoed by weak inception scores (weaker than any in TAB0): 1.07 ± 0.05, 1.29 ± 0.03, 2.28 ± 0.19 for WaveNet ImplA, WaveNet ImplB, and SampleRNN respectively. Note that these inception scores were calculated on far fewer examples (< 1k) than all of the scores listed in TAB0 (which were computed on 50k examples). This is because it took over 24 hours to produce even a thousand one-second examples with these methods (whereas our methods produce 50k examples in a few seconds).Autoregressive methods have not been demonstrated capable of learning to synthesize coherent words without conditioning on rich linguistic features. We are not claiming that these methods cannot learn to synthesize full words, merely that three open-source implementations were unable to do so with default parameters. We want to be clear that our intent is not to disparage autoregressive waveform methods as these methods were developed for a different task, and hence we excluded these poor scores from our table to avoid sending the wrong message. Instead, we hope to highlight that these implementations produced that were noncompetitive for our problem domain, and less useful (due to slowness and lack of a latent space) for creative generation of sound effects. Non GAN-activated cat WaveGAN activated cat SpecGAN activated cat As our improved throughout the course of this research, our cats became quite intrigued by the synthetic bird vocalizations produced by WaveGAN FIG4 ). While this was of course not a formal experiment, we did find this to be encouraging evidence that our method might be capable of producing audio that could additionally convince non-human animals. (n, 1024, 2d) Phase Shuffle (n = 2) (n, 1024, 2d) Conv1D (Stride=4) (25, 2d, 4d) (n, 256, 4d) LReLU (α = 0.2) (n, 256, 4d) Phase Shuffle (n = 2) (n, 256, 4d) Conv1D (Stride=4) (25, 4d, 8d) (n, 64, 8d) LReLU (α = 0.2) (n, 64, 8d) Phase Shuffle (n = 2) (n, 64, 8d) Conv1D (Stride=4) (25, 8d, 16d) (n, 16, 16d) LReLU (α = 0.2) (n, 16, 16d) Reshape (n, 256d) Dense (256d, 1) (n, 1) In TAB2, we list the full architectures for our WaveGAN generator and discriminator respectively. In TAB4, we list the same for SpecGAN. In these tables, n is the batch size, d modifies model size, and c is the number of channels in the examples. In all of our experiments in this paper, c = 1. All dense and convolutional layers include biases. No batch normalization is used in WaveGAN or SpecGAN. In TAB5, we list the values of these and all other hyperparameters for our experiments, which constitute our out-of-the-box recommendations for WaveGAN and SpecGAN. Adam (α = 1e−4, β1 = 0.5, β2 = 0.9)

Represent the TLDR for finding the corresponding scientific article.

Learning to synthesize raw waveform audio with GANs

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.


Question: Represent the article for finding a single-sentence TLDR

Question: The difficulty of obtaining sufficient labeled data for supervised learning has motivated domain adaptation, in which a classifier is trained in one domain, source domain, but operates in another, target domain. Reducing domain discrepancy has improved the performance, but it is hampered by the embedded features that do not form clearly separable and aligned clusters. We address this issue by propagating labels using a manifold structure, and by enforcing cycle consistency to align the clusters of features in each domain more closely. Specifically, we prove that cycle consistency leads the embedded features distant from all but one clusters if the source domain is ideally clustered. We additionally utilize more information from approximated local manifold and pursue local manifold consistency for more improvement. Results for various domain adaptation scenarios show tighter clustering and an improvement in classification accuracy. Classifiers trained through supervised learning have many applications , but it requires a great deal of labeled data, which may be impractical or too costly to collect. Domain adaptation circumvents this problem by exploiting the labeled data available in a closely related domain. We call the domain where the classifier will be used at, the target domain, and assume that it only contains unlabeled data {x t}; and we call the closely related domain the source domain and assume that it contains a significant amount of labeled data {x s, y s}. Domain adaptation requires the source domain data to share discriminative features with the target data . In spite of the common features, a classifier trained using only the source data is unlikely to give satisfactory in the target domain because of the difference between two domains' data distributions, called domain shift . This may be addressed by fine-tuning on the target domain with a small set of labeled target data, but it tends to overfit to the small labeled dataset . Another approach is to find discriminative features which are invariant between two domains by reducing the distance between the feature distributions. For example, domain-adversarial neural network (DANN) achieved remarkable using generative adversarial networks (GANs) . However, this approach still has room to be improved. Because the classifier is trained using labels from the source domain, the source features become clustered, and they determine the decision boundary. It would be better if the embedded features from the target domain formed similar clusters to the source features in class-level so that the decision boundary does not cross the target features. Methods which only reduce the distance between two marginal distributions bring the features into general alignment, but clusters do not match satisfactorily, as shown in Fig. 1(a). As a consequence, the decision boundary is likely to cross the target features, impairing accuracy. In this work, we propose a novel domain adaptation method to align the manifolds of the source and the target features in class-level, as shown in Fig. 1(b). We first employ label propagation to evaluate the relation between manifolds. Then, to align them, we reinforce the cycle consistency that is the correspondence between the original labels in the source domain and the labels that are propagated from the source to the target and back to the source domain. The cycle consistency draws features from both domains that are near to each other to converge, and those that are far apart to diverge. The proposed method exploits manifold information using label propagation which had not been taken into account in other cycle consistency based methods. As a , our approach outperforms other baselines on various scenarios as demonstrated in Sec. 4. Moreover, the role of cycle consistency is theoretically explained in Sec. 3.2 that it leads to aligned manifolds in class-level. To acquire more manifold information within the limited number of mini-batch samples, we utilize local manifold approximation and pursue local manifold consistency. In summary, our contributions are as follows: • We propose a novel domain adaptation method which exploits global and local manifold information to align class-level distributions of the source and the target. • We analyze and demonstrate the benefit of the proposed method over the most similar baseline, Associative domain adaptation (AssocDA) . • We present the theoretical on why the proposed cycle consistency leads to class-level manifold alignment, bringing better in domain adaptation. • We conduct extensive experiments on various scenarios and achieve the state-of-the-art performance. Unsupervised Domain Adaptation It has been shown that the classification error in the target domain is bounded by that in the source domain, the discrepancy between the domains and the difference in labeling functions. Based on this analysis, a number of works have endeavored to train domain-confusing features to minimize the discrepancy between the domains (; ; ; 2017). Maximum mean discrepancy can be used as a measure of domain discrepancy. In an approach inspired by GANs, a domain confusion can be converted into a minmax optimization. While minimization of domain discrepancy can be effective in reducing the upper bound on the error, it does not guarantee that the feature representation in the target domain is sufficiently discriminative. To address this issue, several techniques had been proposed. Explicit separation of the shared representation from the individual characteristics of each domain may enhance the accuracy of the model . This approach has been implemented as a network with private and shared encoders and a shared decoder. The centroid and prototype of each category can be used for class-level alignment . An alternative to such featurespace adaptation techniques is the direct conversion of target data to source data (; ;). Those proposed methods intend to transfer the style of images to another domain while preserving the content. This performs well on datasets containing Figure 2: Overview of our method. The feature generator G projects the input data into the feature space. The dashed line means weight sharing. The embedded source features f s and the target features f t are organized into a graph and then used together to evaluate cycle consistency through label propagation. The embedding classifier C learns from the source ground-truth labels. The discriminator D determines whether features originated in the source or the target domain. images that are similar at the pixel-level; they are problematic when the mapping between high-level features and images is complicated . Metric Learning Metric learning is learning an appropriate metric distance to measure the similarity or dissimilarity between data . Reducing the distances between similar data and increasing the distances between distinct data has shown to improve the accuracy of a classifier. Metric learning is particularly beneficial when very little labeled data is available, which is the situation for domain adaptation. combined metric learning and unsupervised domain adaptation with the enforcement of cycle consistency. In particular, the inner products of source features and target features with the same label are maximized, and minimized between features with different labels. AssocDA enforces the feature alignment between the source and target by forcing the two step round trip probability to be uniform in the same class and to vanish between different classes. Graph-based learning is closely related to metric learning, in that it achieves clustering using distance information. Label consistency is usually assumed, meaning that adjacent data tend to have the same labels . Label propagation has improved the performance of semi-supervised learning by enforcing label consistency by propagating labels from labeled to unlabeled data. To overcome need for fixed graphs to be provided in advance, the distances between each node can be adaptively learned , as in metric learning, and this increases accuracy in both semi-supervised and few-shot learning. Our algorithm, shown in Fig. 2, uses label propagation and cycle consistency to learn features from the source and the target domains which are both 1) indistinguishable each other and 2) close when placed within the same class, but distant when placed in different classes. The details are as follows. Manifold learning extracts intrinsic structures from both unlabeled and labeled data. We obtain these structures by constructing a graph whose vertexes are the embedded features and whose edges are the relations between data. We first embed the input data in the feature space, using the feature generator composed of convolutional layers following previous work . Subsequently, a fully connected graph is constructed according to the distances between the features. The edge weights W ij between the input data x i, x j are determined from the feature vectors using Gaussian similarity, ), where f i, f j are the embedded feature vectors of x i, x j, and σ is a scale parameter. It is known that graph-based methods are sensitive to the scale parameter σ. A large σ in an uniformly connected graph that disregards the latent structure of the data, while a small σ produces a sparse graph which fails to express all the relationship between the data. To adapt σ according to the embedded features, we take σ as a trainable variable to be learned during training. Label propagation is a method of manifold regularization, which in turn produces a classifier that is robust against small perturbations. Label propagation can be seen as a repeated random walk through the graph of features using an affinity matrix to assign the labels of target data . A label matrix y n ∈ R (Ns+Nt)×C refers to the labels assigned to data in both domains at the n-th step random walk. The dimension of y n is determined by N s, N t, and C which are the numbers of source and target data points and the number of classes, respectively. The first N s rows of y n contain the labels of the source data, and the remaining N t rows contain the labels of the target data. The initial label vector y 0 contains y s for the source data, which is one-hot coded ground-truth labels and zero vectors for the target data. The one step of the random walk transforms the label vector as follows: where,. W ts is a similarity matrix between the target and source data, and W tt is a similarity matrix which represents the interrelations in the target data. These are described in the Sec. 3.1. The normalization operation normalize(·) transforms the sum of each row to 1. The identity matrix in the normalized transition matrix T signifies that the labels of source data do not change because its labels are already known. In graph theory, these source data points would be called absorbing nodes. In label propagation, the labels of the target domain is assigned to the propagated labelsŷ t by infinite transition, formulated asŷ tt T ts y s, which converges as follows In our method,ŷ t is used to obtain the propagated labels of the source data in the same way aŝ y s = (I − T ss) −1 T stŷ t where T ss and T st are defined analogous to T tt and T ts, so that we can learn the features of which clusters match each other. We then refer to the property thatŷ s should be the same as the original label y s as cycle consistency. Pursuing cycle consistency forces not perfectly aligned features to move toward the nearest cluster, as shown in Fig. 3. The following theorem shows that enforcing cycle consistency on ideally clustered source data will segregate different classes of the source and the target data and gather the same classes. Theorem 1. Let {e i |1 ≤ i ≤ C} be the standard bases of C-dimensional Euclidean space. For the sake of simplicity, source data x 1, x 2, · · ·, x Ns are assumed to be arranged so that the first n 1 data belong to class 1, the n 2 data to class 2, and so forth. Assume that 1) the source data is ideally clustered, in the sense that T ss has positive values if the row and the column are the same class and zero otherwise, i.e., T ss = diag(T 1, T 2, · · ·, T C), the block diagonal where T i is a n i × n i positive matrix for i = 1, 2, · · ·, C and 2)ŷ s = y s. Then for all 1 ≤ j ≤ C, there exists a nonnegative vector v j ∈ R Ns such that 1) the part where source data belongs to j th class (from th element to [n 1 + n 2 + · · · + n j] th element) are positive and the other elements are all zero and 2) v j T stŷ t e i = 0 for all 1 ≤ i ≤ C, i = j. Proof. The illustration and the proof is given in Appx. A. In Thm. 1,ŷ t e i refers to the assigned probability as i th class to the target data. The implies that if a target data is enough to be predicted as i th class through label propagation, i.e., i th elements of the row inŷ t corresponding to the target data is nonzero, then the elements of T st which represent the transitions from source data of all but i th class to the target data should vanish, i.e., the target data is segregated from the source data in different classes. As described in Sec. 3.4, we employed DANN to prevent the target data distribution to be distinct from the source data distribution. If a column of T st is a zero vector, the feature of the corresponding target data for the column is considerably distant from all source data features. However, minimizing the DANN loss makes target features lie around source features, and thus each column of T st is not likely to be a zero vector. Combining this conjecture with Thm. 1, each row ofŷ t has only one nonzero value, i.e., every target data belongs to only one cluster. We thus argue that by pursuing this property, generator can learn more discriminative shared features, and classification performance may improve. Cycle consistency is enforced by minimizing the l 1 loss L cycle betweenŷ s and y s: Comparison with AssocDA The proposed method has some resemblance with AssocDA in that they both consider the similarities and transitions between data. However, we argue that AssocDA is a special case of our method. First, our method exploits manifold over each domain by taking relations within the same domain into account through label propagation, whereas AssocDA only considers relations across the domains. Specifically, in Eq. 1, our method utilizes both T ts and T tt, but AssocDA ignores T tt which often has useful information about the target data manifold. Second, AssocDA forces the two-step transition to be uniform within the same class. This strict condition may drive the source features of each class to collapse to one mode and can cause overfitting. On the contrary, our method only constrains source data to preserve its original labels after the label propagation. Thus, it does not require all source data be close to each other within the same class; it allows moderate intra-class variance. The experiment in Sec. 4.1 and Fig. 4 support these arguments and visualize the effect of the differences. As shown in Thm. 1, the introduced cycle consistency utilizes graph based global manifold information and enforces the source and target features to be aligned in class-level. However, in practice, the limited size of mini-batch may restrict the available information of graph. The knowledge from the local manifold of each sample, in this case, can complement the global manifold information. In this regard, we additionally pursue local manifold consistency that the output should not be sensitive to small perturbations in the local manifold, as suggested elsewhere (; ;). Concretely, localized GAN (LGAN) is employed to approximate the local manifold of each data and sample a marginally perturbed image along the local manifold from the given data. LGAN allows it as LGAN focuses on learning and linking patches of local manifolds in its training procedure. The difference between the predicted label of the perturbed image and that of the original image is minimized to impose local manifold consistency of the classifier as follows: where, C, G and G L are the embedding classifier, the feature generator and the LGAN generator, respectively. LGAN generator, G L (x, z), takes an image x and noise z to generate locally perturbed image along the approximated local manifold. H(·, ·) denotes cross entropy. µ and η are coefficients for the source and the target local manifold consistency loss, respectively. Our method learns a clustered feature representation that is indistinguishable across the source and target domains through the training process as follows: where, D is the discriminator. α and β are coefficients for the last two terms and λ is a scheduling parameter described in Appx B.1. L class is a widely used cross-entropy loss for labeled source data and L dann is a GAN loss : where discriminator's output D(·) is the probability that the input originated from the source domain. From the metric learning perspective, L class serves to separate the source features according to their ground-truth labels, which supports the assumption in Thm. 1, the ideally clustered source features. Subsequently, L dann takes a role in moving the target features toward the source features, but it is insufficient to produce perfectly aligned clusters. Our cycle loss L cycle and local loss L local facilitate clustering by enforcing cycle consistency and local manifold consistency. We present a toy example to empirically demonstrate the effect of our proposed cycle loss using manifold information compared to the most similar method, AssocDA. We designed synthetic dataset in 2-dimensional feature space with two classes as illustrated in the leftmost of Fig. 4. The source data lie vertically and the target data are slightly tilted and translated. The second column shows the negative gradients of AssocDA loss and our cycle loss with respect to each data. Negative gradients can be interpreted as the movement of features at each iteration. The third and fourth are the updated features using gradient descent in the middle and at the end of feature updates 1. As argued in Sec. 3.2, AssocDA does not consider the transition within the same domain and thus target data which are close to source data with different label (points inside red circles in the second column) are strongly attracted to them. On the other hand, the gradients of the cycle loss are much smaller than AssocDA. We speculate that it is because the attractions from source data in the same class are propagated through target data manifold. As a , AssocDA leads some data to move in wrong direction, being misclassified, while cycle loss brought correctly aligned manifolds. In addition, AssocDA attracts all features too close at the end of updates, which may cause overfitting. Last but not least, our cycle loss aligned source and target clusters correctly without the aid of dann loss. We thus argue that our method is complementary to DANN rather than an extension. We show the performance of the proposed method on two real visual dataset. First dataset, which we call by Digit & Object dataset, includes digit dataset such as SVHN and Synthetic Digits (DIGITS), Gradients of loss Progress at 150 steps Progress at 600 steps Figure 5: Visualization of learned features using t-SNE. Circles and x markers respectively indicate the source and target features. Colors correspond to labels. In all cases, the features from two domains form similar and tight clusters, which is the key objective of our method. and object dataset such as STL and CIFAR. We used ImageCLEF-DA as second dataset for more challenging benchmark. We employed three networks as previous work (; ; . A network with two convolutional layers and two fully connected layers for digit dataset and a network with nine convolutional layers and one fully connected layer for object dataset were implemented. Pretrained ResNet was used for ImageCLEF-DA dataset. More details on training settings, adaptation scenarios and an experiment on non-visual dataset are provided in Appx. B.1, B.2 and E. Tab. 1 compares the accuracy of our method on Digit & Object dataset with that of other approaches. For our method, we reported the of three models, one with local loss (L), another with cycle loss (C) and the other with both losses (C+L). Our algorithm outperformed the others on most of the tasks. In the most experiments, the performance of the proposed method was better than the state-ofthe-art. This suggests that enforcing alignment in addition to domain-invariant embedding reduces the error-rate. PixelDA showed superior performance on MNIST→MNIST-M, but it is attributable to the fact that PixelDA learns transferring the style of images at a pixel level which is similar to the way MNIST-M is generated from MNIST. T-SNE embeddings in Fig. 5 indicates that the learned features are well aligned and clustered. Tab. 2 reports the on ImageCLEF-DA dataset experiments. The performance of our method was better than or comparable to those of other baselines. Especially, our method outperforms which also aims to learn clustered and aligned features. Although the objectives are related, the approaches are quite different. Our method utilizes the manifolds of the source and the target domain through label propagation and cycle consistency, whereas CAT considers the distance between two samples for clustering and the distance between the first-order statistics of distributions for alignment. We argue that the better performance is attributed to utilizing manifold information beyond one to one relations of which benefits are explained in Sec. 4.1. Throughout ImageCLEF-DA experiments, the proposed method without the local loss achieved better accuracy compared to that with the local loss. Approximation of the local manifold on ImageCLEF-DA generated by LGAN was slightly worse than that on Digit & Object dataset; perturbed image was blurred and semantically invariant with the original image. Hence, we speculate that the performance of the proposed method may be improved with better local manifold approximation. In this paper, we proposed a novel domain adaptation which stems from the objective to correctly align manifolds which might in better performance. Our method achieved it, which was supported by intuition, theory and experiments. In addition, its superior performance was demonstrated on various benchmark dataset. Based on graph, our method depends on how to construct the graph. Pruning the graph or defining a similarity matrix considering underlying geometry may improve the performance. Our method also can be applied to semi supervised learning only with slight modification. We leave them as future work. A PROOF OF THEOREM 1 Theorem 1. Let {e i |1 ≤ i ≤ C} be the standard bases of C-dimensional Euclidean space. For the sake of simplicity, source data x 1, x 2, · · ·, x Ns are assumed to be arranged so that the first n 1 data belong to class 1, the n 2 data to class 2, and so forth. Assume that 1) the source data is ideally clustered, in the sense that T ss has positive values if the row and the column are the same class and zero otherwise, i.e., T ss = diag(T 1, T 2, · · ·, T C), the block diagonal where T i is a n i × n i positive matrix for i = 1, 2, · · ·, C and 2)ŷ s = y s. Then for all 1 ≤ j ≤ C, there exists a nonnegative vector v j ∈ R Ns such that 1) the part where source data belongs to j th class (from [n 1 + n 2 + · · · + n j−1 + 1] th element to [n 1 + n 2 + · · · + n j] th element) are positive and the other elements are all zero and 2) v j T stŷ t e i = 0 for all 1 ≤ i ≤ C, i = j. From the assumption, T ss is a block diagonal matrix of which block elements are T 1,T 2,· · ·,T C. v j is all zero except n j elements in the middle of v j. The n j elements are all positive and their indices correspond to those of T j in T ss. In the proof, the left eigenvector u j of T j will be substituted to this part. Proof. From the Perron-Frobenius Theorem that positive matrix has a real and positive eigenvalue with positive left and right eigenvectors, T j, the block diagonal element of T ss, has a positive left eigenvector u j with eigenvalue λ j for all j = 1, 2, · · · C. Then, as shown below, v j = (0 0 ··· 0 u j 0 ··· 0) where n 1 + n 2 + · · · + n j−1 zeros, u j and n j+1 + n j+2 + · · · + n C zeros are concatenated, is a left eigenvector of T ss with eigenvalue λ j by the definition of eigenvector. From the label propagation, we have,ŷ By multiplying v j (I − T ss) on the left and e i on the right to the both sides in Equation 13 and combining with the assumptionŷ s = y s, we have, The last zero comes from the definition of v j. In this subsection, we offer the modified version of Thm. 1 when the source features are slightly perturbed from the ideally clustered condition and the other assumption y s =ŷ s holds. We start from representing T ss as follows to indicate the perturbation. where, δT ss is assumed to be sufficiently small under infinite norm and T ss is a block diagonal transition matrix when the source features are ideally clustered as stated in Thm. 1. In the proof above, we showed eigenvalue λ j and its corresponding eigenvector v j of T j. According to perturbation theory of eigenvalue and eigenvector , the eigenvector can be approximated by first order when the perturbation is small. More generally and precisely, where, the norm is vector or matrix 2-norm and m j is determined by T ss. For the sake of simplicity, we use Big-O notation in Eq. 19 and Eq. 20. Now, we reuse Eq. 16 from the proof of Theorem 1 since it is still valid under the modified condition. We apply Eq. 19 to the right hand side as follows, where i = j. Eq. 24 holds because only j th block elements of v j are nonzero. We also used the fact that y s is bounded by 0 and 1. Similarly, the left hand side of Eq. 22 can be transformed as follows, The second term of Eq. 26 holds because T st andŷ t are bounded by 0 and 1. Finally, by combining Eq. 24 and Eq. 26, we have, Eq. 27 implies that if the perturbation is sufficiently small i.e., ||δT ss || ∞ << 1 and a target data is enough to be predicted as i th class through label propagation, then the transitions from source data of all but i th class to the target data is negligible because v j is positive for j th block and zero for others. It is the same with the of Theorem 1. In addition, the more strongly the target data is classified as i th class i.e., the corresponding element ofŷ t becomes greater, the smaller the transitions from source data in the other classes are, indicating the segregation against the other classes. Practically, the coefficients for L cycle and L cycle are scheduled to facilitate the clustering of source features correctly at the early stage of training. Thus we may assume that T ss is marginally perturbed around the ideally clustered one when our cycle loss takes effect. Scheduling the effect of losses To reduce the effect of noisy signal from L dann and L cycle during the early stages of training, a weight balance factor λ = 2 1+exp(−γ·p) − 1 is applied in Eq. 6. A constant γ determines the rate of increase of λ; p is the progress of training, which proceeds from 0 to 1. The parameter was introduced to make a classifier less sensitive to the erroneous signals from the discriminator in the beginning. Throughout the experiments, γ was set to 10. Hyperparameter Although it would be ideal to avoid utilizing labels from the target domain in the hyperparameter optimization, it seems that no globally acceptable method exists for this. One possibility is reverse validation scheme but this may not be accurate enough to estimate test accuracy . In addition , applications exist where the labeled target domain data is available at the test phase but not at the training phase. Hence, we adopted the protocol of that exploits a small set of labeled target domain data as a validation set; 256 samples for the Amazon review experiment and 1,000 samples for the other experiments (; 2017;). During training, Adam optimizer with learning rate of 10 −3 was utilized. Exponential moving averaging was applied to the optimization trajectory. It is an inherent characteristic of our method that each data sample affects the graph structure. So it is important for each class sample in each batch to represent its classes accurately. In other words, the transition matrix can be corrupted by biases in the samples. Therefore, the number of data samples in each class in a batch should be sufficient to avoid any likely bias. To address this problem, we performed experiments with batch size of up to 384 and observed very little improvement beyond a batch size of 128. So we fixed the batch size to 128 for Digit & Object dataset. For the ImageCLEF-DA dataset, we set the batch size to 36 because of limited computing resource. MNIST → MNIST-M The MNIST database of hand-written digits consists of digit images with 10 classes and MNIST-M consists of MNIST digits blended with natural color patches from the BSDS500 dataset . In addition, following other work the colors of the MNIST images were inverted randomly, because their colors are always white on black, whereas the MNIST-M images exhibit various colors. MNIST ↔ USPS USPS is another dataset of hand-written images of digits, with 10 classes. USPS contains 16×16 images and the size of the USPS image is upscaled to 28×28, which is the size of the MNIST image in our experiment. The evaluation protocol of CYCADA is adopted. SVHN → MNIST The Street View House Numbers (SVHN) dataset consists of images of house numbers acquired by Google Street View. The natural images that it contains, are substantially different from the line drawings in the MNIST dataset. The size of each MNIST image is upscaled to 32×32, which is the size of SVHN images. SYN DIGITS → SVHN SYN DIGITS dataset is synthetic number dataset which is similar to the SVHN dataset . The most significant difference between the SYN DIGITS dataset and the SVHN dataset is the untidiness in the of real images. CIFAR ↔ STL Both CIFAR dataset and STL dataset are 10-class datasets that contain images of animals and vehicles. Not overlapped classes are removed to make a 9-class domain adaptation task . We used the larger network only for this experiment. 2 The twelve common classes of three publicly available dataset (Caltech-256, ImageNet ILSVRC2012, and PASCAL VOC2012) are selected to form visual domain adaptation tasks. We perform all six possible adaptation scenarios among these three dataset. We searched hyperparameters within α = {0, 0.01, 0.1, 1}, β = {0.01, 0.1, 1}, µ = {0, 0.01} and η = {0, 0.1}. Perturbation to the LGAN generator, i.e. z, is fixed to 0.5 for all experiments. The best hyperparameters for each task is shown in Table. 3. Setting an appropriate value for the scale parameter, σ, is important because it has a substantial role in determining the transition matrix, T. Therefore, we conducted several experiments with fixing σ to various values. For these experiments, we excluded L local to observe the effect of σ.' Adapt' means that the σ is learned to adapt according to the embedded features. For four out of five scenarios, fixing σ to 1 performed better than fixing it to 0.1 or 10. With this observation, we initialized σ to 1 took it as a trainable variable. The of adaptively learning σ is reported at the bottom row of the table. Compared to fixing σ to 1, adaptively learning σ achieved better accuracy and had a lower standard deviation range which means that it is more stable. We also would like to highlight that our model is robust to the initial value of σ. We conducted extensive experiments with initializing σ to 0.1, 1 and 10 and taking it as a trainable variable. Except for SVHN → MNIST transfer task with setting initial σ value to 10, the initial value of σ has a minute influence to the accuracy. We believe that adaptively learning the scale parameter can be usefully employed in any other graph-based method. The learned σ values for various scenarios are as follows. It seems that σ adaptively learns its value according to the transfer task, regardless of its initialization. We tried l 2 loss and cross entropy loss to enforce cycle consistency as well. We excluded L local to compare the effectiveness of these functions. For all Digit dataset adaptation experiments, evaluating cycle consistency with l 1 norm achieved the highest accuracy. We speculate that l 1 norm is more numerically stable or provides more effective gradients than other functions in this case. The Amazon Reviews dataset provides a non-visual domain for domain adaptation experiments. It contains reviews of books, DVDs, electronics, and kitchen appliances encoded as 5,000-dimensional feature vectors containing unigrams and bigrams of the texts with binary labels. Four-and five-star reviews are labeled'positive'; reviews with fewer stars are labeled'negative'. We used 2,000 labeled source data and 2,000 unlabeled target data for training, and between 3,000 to 6,000 target data for testing. Tab. 8 shows that our method performs better than DANN , VFAE and ATT on the Amazon Reviews data in six out of twelve experiments. Our method was more accurate than DANN in nine out of twelve settings, showing approximately 2.0% higher classification accuracy on average.

Represent the TLDR for finding the corresponding scientific article

A novel domain adaptation method to align manifolds from source and target domains using label propagation for better accuracy.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a method based on the adversarial training strategy to learn discriminative features unbiased and invariant to the confounder(s) by incorporating a loss function that encourages a vanished correlation between the bias and learned features.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We propose a new method for training neural networks online in a bandit setting. Similar to prior work, we model the uncertainty only in the last layer of the network, treating the rest of the network as a feature extractor. This allows us to successfully balance between exploration and exploitation due to the efficient, closed-form uncertainty estimates available for linear models. To train the rest of the network, we take advantage of the posterior we have over the last layer, optimizing over all values in the last layer distribution weighted by probability. We derive a closed form, differential approximation to this objective and show empirically over various models and datasets that training the rest of the network in this fashion leads to both better online and offline performance when compared to other methods. Applying machine learning models to real world applications almost always involves deploying systems in dynamic, non-stationary environments. This dilemma requires models to be constantly re-updated with new data in order to maintain a similar model performance across time. Of course, doing this usually requires the new data to be relabeled, which can be expensive or in some cases, impossible. In many situations, this new labeled data can be cheaply acquired by utilizing feedback from the user, where the feedback/reward indicates the quality of the action taken by the model for the given input. Since the inputs are assumed independent, this task can be framed in the contextual bandit setting. Learning in this setting requires a balance between exploring uncertain actions (where we risk performing sub optimal actions) and exploiting actions the model is confident will lead to high reward (where we risk missing out on discovering better actions).Methods based on Thompson sampling (TS) or Upper Confidence Bounds (UCB) provide theoretically BID1 BID0 and empirically established ways BID12 BID3 for balancing exploration/exploitation in this setting. Unfortunately, both methods require estimation of model uncertainty. While this can be done easily for most linear models, it is a difficult and open problem for large neural network models underlying many modern machine learning systems. An empirical study by BID17 shows that having good uncertainty estimates is vital for neural networks learning in a bandit setting. Closed formed uncertainty estimations (and online update formulas) are available for many linear models. Since the last layer of many neural networks are usually (generalized) linear models, a straightforward way for learning neural networks in a bandit setting is to estimate the uncertainty (as a distribution over weights) on the last layer only, holding the previous layers fixed as feature functions which provide inputs to the linear model. This method (and variants thereof) has been proposed in bandit settings BID17 BID13 as well as other related settings (; BID16 BID11 and has been shown to work surprisingly well considering its simplicity. This style of methods, which we refer to as Bayesian last layer or BLL methods, also has the advantage of being both relatively model-agnostic and scalable to large models. Of course, BLL methods come with the tacit assumption that the feature functions defined by the rest of the network output good (linearly separable) representations of our inputs. This means that, unless the input data distribution is relatively static, the rest of the network will need to be updated in regular intervals to maintain low regret. In order to maintain low regret, the retraining objective must: 1) allow new data to be incorporated quickly into the learned model, and 2) prevent previously learned information from being quickly forgotten. Previous papers retrain BLL methods simply by sampling minibatches from the entire pool of previously seen data and maximizing log-likelihood over these minibatches, which fails to meet the first criteria above. In this paper we present a new retraining objective for BLL methods meeting both requirements. We avoid retraining the last layer with the entire network (throwing out the uncertainty information we learned about the last layer) or retraining with the last layer fixed (fixing the last layer to the mean of its distribution). Instead, we utilize the uncertainty information gathered about the last layer, and optimize the expected log-likelihood of both new and old data, marginalizing 1 over the entire distribution we have on the last layer. This gives a more robust model that performs relatively well over all likely values of the last layer. While this objective cannot be exactly computed, we derive a closed form, differentiable, approximation. We show that this approximation meets both criteria above, with a likelihood term to maximize that depends only on the new data (meeting the first point), and a quadratic regularization term that is computed only with previously seen data (meeting the second point). We show empirically that this method improves regret on the most difficult bandit tasks studied in BID17. We additionally test the method on a large state-of-the-art recurrent model, creating a bandit task out of a paraphrasing dataset. Finally, we test the method on convolutional models, constructing a bandit task from a benchmark image classification dataset. We show that our method is fast to adapt to new data without quickly forgetting previous information. Contextual bandits are a well researched class of sequential decision problems (; BID10, of which, many variants exist. In this paper, we mainly consider the multiclass contextual bandit problem studied in BID7 . The problem takes place over T online rounds. On round t, the learner receives a context x t, predicts a class label y t, and receives binary reward r t indicating whether the chosen label is correct. No other information is received about the other classes not picked. In our setting, we assume each class c (the arms of the bandit) has associated with it a vector z c and that the probability of a class label is modeled by: DISPLAYFORM0, where σ is the logistic function and f θ is a feature function parameterized by θ. In our case, f θ defines a neural network, while z c can be seen as the last layer of the network 2.Our goal is to get low regret, R = T i r * i − T i r i, where r * i is the optimal reward at step i. The key to getting low regret is employing a policy for balancing exploration and exploitation. If we capture the uncertainty in each z c at time t by modeling its posterior distribution over previous data D t−1 as a multivariate Gaussian, z c ∼ p(z c |D t−1), then we can easily deploy sound exploration strategies such as Thompson sampling or UCB. If we hold f θ fixed, then we can easily model this distribution by doing an online Bayesian regression on the outputs of f θ, which gives us closed form formulas for updating the posterior over the last layer (specifically, its mean and covariance) given a single datapoint.3 When f θ is a neural network, then this is an instance of a BLL method. BLL methods have been shown to be an effective, model agnostic, and scalable way to deal with exploration problems involving neural networks. Previous work has found them to be a pragmatic method for obtaining approximate uncertainty estimates for exploration BID13 BID17 BID16 BID2 and as proxies for Gaussian processes in both Bayesian Optimization problems and general regression tasks BID11.If f θ is fixed, then z c can be updated efficiently in an online manner. An unanswered question still remains however: how does one actually update and learn f θ? If you don't care about achieving low regret (ie you only care about offline performance), then the answer is easy; just gather your data, train f θ offline, possibly with off-policy learning methods (; BID6, and learn the Bayesian regression post-hoc. Of course, if you are concerned about online performance (regret) then this is not a viable option. A training method for f θ must take care of two things: when do we update the feature functions and what do we update them with? A reasonable answer to the former question is to update on a fixed schedule (every T rounds). In this paper, we focus on answering the latter questions of which there are two obvious solutions, each with a corresponding problem: Sample minibatches only from recent data. Problem: We may overfit on this data and forget old information. Sample minibatches uniformly from the set of all collected data (both recent and old). Problem:We have lost the ability to adapt quickly to new data. If the input distribution suddenly shifts, we will likely have to wait many iterations before newer data becomes common enough in our minibatch samples, all while our regret increases. One thing to consider is that when it comes time to train our feature functions, we have access to a distribution over our last layer. If, for example, our distribution has suddenly shifted, then the last layer distribution should have more variance, ideally placing density on last layers that do well on older data, as well as those that fit well to the new data. If the distribution remains the same, then the variance should be low and density should be placed on relatively few values. Intuitively, we can get the best of both worlds (ability to adapt or retain information when needed) by optimizing over all values of the last layer weighted by their probability. In the next section, we derive a local approximation to this objective that shows this indeed is the case. Let D t and D t−1 be the most recent set of data collected online, and the set of all previously collected data, respectively. Additionally, assume a zero mean Gaussian prior over the last layer, p(z|θ) that is constant with respect to θ. Recall that during online training we fix the parameters θ = θ t−1, and model the distribution Q = p(z c |D t, D t−1, θ t−1). We want a value of θ such that both our data and the last layer values drawn from Q are likely. Thus our objective is to maximize: DISPLAYFORM0 We can write the marginal likelihood as a convolution between a logistic function and a Gaussian (based on our assumption of zero mean Gaussian prior p(z|θ)), which can be approximated in closed form as per MacKay FORMULA1: DISPLAYFORM1 Where µ is the mean of p(z|θ). Since we have a zero mean prior, the above term evaluates to σ whose value is a constant 1 2. Using this , we can rewrite equation FORMULA1 as: DISPLAYFORM2 Where c is a constant entropy term which can be ignored. For the first expectation term, we can use a second order Taylor expansion around E z∼Q [log p(D t |z, θ)] to get a closed form approximation 4: DISPLAYFORM3 This approximation was used in and shown to work well empirically. The expectations in equation FORMULA4 The KL term in equation FORMULA3 can also be approximated locally with a second Taylor expansion around the current value of θ = θ t−1. Let K(θ) = KL(Q||p(z|D t−1, θ)). Then, the second order Taylor expansion around θ = θ t−1 is: DISPLAYFORM4 Utilizing properties of KL divergence, as well as equation FORMULA2, it can be derived 5 that K (θ t−1) will evaluate to 0, and K (θ t−1) will evaluate to βF P, where β = DISPLAYFORM5 and F P is the Fisher Information Matrix of P = p(z|D t−1, θ t−1). Getting rid of constants, we can write the local KL approximation (when θ is close to θ t−1) as: DISPLAYFORM6 The term β defines the ratio between the expected data likelihood given the last layer z distributed as z ∼ p(z|D t−1, θ t−1) and the expected data likelihood given the last layer is distributed under the prior p(z|θ). This indicates that the better our previous values of θ t−1 and z explain the data, the more we should regularize when incorporating new data (i.e raising the value of β). In practice, these values may be computed or approximated, however for efficiency, we treat β as a hyperparameter, and linearly increase it throughout the learning process. Our final objective to optimize is thus: DISPLAYFORM7 Notice that the old data is now only used to calculate the Fisher information matrix F P and is not actually involved in the optimization. Thus, the optimization (at least temporarily) over all our data can be done by simply drawing minibatches from the new data only, while the old data is only used to calculate the regularization term. The practical benefit of this is that the regularization term can be easily computed in parallel while doing the online Bayesian regression and collecting new data. The quadratic regularization term shares similarities to objective functions in continual learning which aim to prevent catastrophic forgetting BID8 ). Combining the online learning and the network retraining stages described in the previous section gives us the general form of the iterative algorithm we study in this paper. The algorithm alternates between two stages:Online Phase: As input, take in a set of data D t−1, the posteriors (one for each arm) of the last layer conditioned on previous data p(z|D t−1) as well as a fixed value of f θ. This phase takes place over a series of T online rounds. In every round, the learner receives a context x i, and uses the posteriors over the last layer to decide which arm to pull (via Thompson sampling or UCB). The learner receives feedback y i upon pulling the arm c, and updates the posterior over z c with it. After T rounds, the learner outputs the updated posteriors over z, and the data collected this phase, D t.Offline/Retraining Phase: As input, take in D t, D t−1, and the posteriors over z. Retrain f θ using method described in Section 3. Set D t−1 = D t ∪D t−1. Recompute the posteriors over z conditioned on D t−1 using the new value of f θ. Output D t−1, f θ, and the updated posteriors p(z|D t−1).The marginalization method described in Section 3 is one type of retraining method. We compare it against two other methods in the next section and present for various experiments. We evaluate our technique across a diverse set of datasets and underlying models. As an initial sanity check, we first evaluate our method on the three most difficult (but low dimensional) bandit tasks analyzed in BID17. We next look at two higher dimensional problems and models; one being a Natural Language Processing (NLP) task using a state-of-the-art recurrent model and the other being a vision task using a convolutional model. In particular, we look at the degree at which each method can achieve both good online performance (regret) and good offline performance (offline test set accuracy), even in the face of large shifts in the data distribution. We provide additional details on hyperparameters, experimental setup, and dataset information in Appendix 7.4. All experiments are run 5 times, and reported with the mean and standard error. We evaluate all the datasets against the baseline presented in BID17, as well as a variant of our proposed method. Bandit feedback. In this setting the models are trained using bandit feedback as the label:Marginalize. This is our method of marginalizing over all values of the last layer for the neural network training. Minibatches are sampled from the new data only and the regularization term is computed from the old data. Sample New. This baseline creates minibatches using only the newly collected data, optimizing the likelihood of the new data only. It is equivalent to our method with a regularization constant of zero. As mentioned in Section 2.1, this method is good at adapting to new data but has a drawback of forgetting the old information. Sample All. This is the retraining method presented in BID17. In this method, a set number minibatches are created by uniformly sampling from all collected data (both old and new). SGD gradient updates are then performed using these batches. This method is slow to adapt but retains older information (refer Section 2.1).Full feedback. In this setting models are trained using all the labels for the datasets:Batch Train. When evaluating the offline accuracy, we also give the for a model that has been trained on the shuffled data in batch mode, with all the labels for the dataset (i.e. full feedback). This measures how well we could do given we had access to all the labels (instead of just the bandit feedback), and trained in a normal offline setting. Surprisingly, as we see in some cases, training online with marginalization sometimes performs comparable to training offline. We first confirm that our method gives good online performance on simpler, but previously studied, problems in BID17.We present on the three hardest bandit tasks analyzed in BID17, the Census, Jester, and Adult dataset. The bandit problems for these datasets are defined as in previous work:Census and Adult. Both these datasets are used for multiclass classification problem. Census dataset has 9 classes whereas Adult dataset consists of 14 classes. For both datasets the bandit problem is created as follows: for each class we assign an arm, and each arm is associated with a logistic regression (parametrized by a vector) that takes a context as input and returns the expected reward (0 or 1) for selecting the arm. In the online round we receive a context (feature vector) and pick an arm according to some policy (like UCB), and receive a reward. Only the picked arm is updated in each round. Jester BID5 This dataset consists of jokes with their user rating. For the bandit problem, the model receives a context representing a user along with 8 jokes out of which it is required to make recommendation of 1 joke. In this setting each joke is defined as a bandit arm. The problem here is similar to above with the exception that each arm is associated with a linear regression and outputs the predicted user rating for the selected joke. The reward returned is the actual user rating for the selected joke. As previously done in BID17, we use a two layer MLP as the underlying model, using the same configuration across all methods. For Marginalize and Sample New, we perform the retraining after 1000 rounds. For Sample All we update after 100 rounds just like BID17. In Table 1 we report the average cumulative regret as well as the cumulative regret relative to a policy that selects arms uniformly at random. We report the for both Thompson Sampling (TS) and for UCB policies. Results are similar for either UCB and TS which shows that policies does not influence performance of the training mechanisms. On most of the tasks both Marginalize (our method) and Sample New outperforms Sample All (method used in BID17) in terms of cumulative regret. Both Marginalize and Sample New techniques are very similar in performance for the three datasets. All the three datasets used in this experiment are low dimensional, static, and relatively easy to learn, hence there is not much history to retain for Sample New technique. In the next section we will present on larger datasets and also evaluate where we will show that our method performs better than Sample New. Next we evaluate our method with a bigger and more complex underlying model on the NLP domain. We selected Bilateral Multi-Perspective Matching (BiMPM) , a recurrent model that performs well on several sentence matching tasks, including the paraphrase identification task, to evaluate our method. The goal of the paraphrase identification task is to determine whether a sentence is a paraphrase of another sentence, i.e., whether they convey the same meaning. To evaluate whether our algorithm is robust to shifts in data distribution we combined two different paraphrase identification datasets: i) The Quora Question Pairs dataset (Quora), 6 which contains 400,000 question pairs from the QA website Quora.com, and ii) The MSR Paraphrase Corpus (MSR) BID4, which contains 5,800 pairs of sentences extracted from news articles. To create an online training dataset we concatenate the MSR training set to a sample of 10,000 examples from the Quora training dataset 7.We run the online algorithms on this dataset to report the regret values, while we report the offline performance on the MSR and Quora test sets. We use UCB as our search strategy, as it performs similarly to Thompson sampling and runs much faster in our implementation. We analyze the following two bandit tasks:Multiclass. Like the previous datasets, we create a bandit problem by treating each class as an arm parameterized by a vector and the contexts as the individual data instances. A reward 1 is awarded for identifying correctly if the two sentences in the pair are paraphrase. For each method, we perform an offline retraining after 1,000 online rounds. Pool. Like the multiclass task, the pool based task occurs over a series of rounds. On each round, the model receives a pool of k(=3) instances, and must select one of them for the user. After that the model receives a reward based on its selection. The goal of the model is to learn a scoring function that predicts the expected reward for selecting a certain instance, while at the same time trying to keep regret low. This setting can be seen as an instance of the bandit problem formulation described in . In our case, our instances are candidate paraphrase pairs, where the model gets a reward of 1 for returning a valid paraphrase pair, and 0 otherwise. We use the same implementation and hyperparameters for BiMPM as in . For Marginalize and Sample All, we perform the retraining every 500 rounds. Sample New performed poorly offline on this setting and is thus updated every 1,000 rounds. In TAB2 we show that our method Marginalize outperforms both Sample All and Sample New techniques for both multiclass and pool based tasks. Sample All and Sample New have comparable cumulative regret. Sample New has worse offline accuracy on Quora dataset (because it forgets old information), while it has better offline accuracy on MSR (because it is able to adapt quicker). For Batch train, both multiclass and pool based tasks are same-a binary classification problem. Batch train performs only slightly better than our method in terms of offline accuracy, where Batch train gets full feedback, while our method only gets partial (bandit) feedback. FIG1 further shows that when the data distribution changes (switching form Quora to MSR in the pool based task) Marginalize and Sample New are able to adapt much faster than Sample All. Overall Marginalize achieved a lower regret as well as higher offline accuracy for both the bandit settings. We additionally evaluate our method using a convolutional neural network, which is a common network architecture for computer vision applications. Table 3: Image classification Bandit Task on CIFAR-10 dataset. We use CIFAR-10 dataset BID9 ) for this experiment. It is a commonly used dataset for image classification task consisting of 60,000 images from 10 different classes. Similar to the Quora/MSR tasks, we simulate a domain shift by concatenating together two datasets. In this case we create two data sets from CIFAR-10 by partitioning the dataset into images depicting animals (6 classes) and images depicting transportation (4 labels). As above, we analyze two bandit tasks:Multiclass. We define the multiclass bandit task similarly as above, for each of the 10 classes in CIFAR-10, we assign one arm. At each round, the model receives an image, guesses a class, and receives feedback (0 or 1) for this class only. The task is considerably more difficult than the multiclass paraphrase bandit due to the number of classes. We use 1,000 rounds for the retraining frequency for all methods. Pool. We also define a pool based bandit, similar to the pool based paraphrase bandit, with pool size k = 5. In this case we turn CIFAR-10 info a binary classification task. We select the two most difficult classes to classify (airplanes and birds, according to the confusion matrix of our base CNN model) in CIFAR-10, and denote these as the positive class. Like the previous pool task, the learner receives a pool of images and must select one. A reward of 1 is given for selecting an image belonging to the positive class, 0 otherwise. As done in the previous pool task, the data is sorted as to simulate a change in domain. We use a standard convolutional neural network architecture for this task, detailed in Appendix 7.5. We use 500 rounds for the retraining frequency for all methods. In Table 3 we present for the image classification bandit task, using average cumulative regret and offline accuracy as evaluation metrics. Again, Sample New performs better than Sample All for cumulative regret but under-performs in the offline setting. As expected, our method performs well for both cumulative regret and offline setting. For the multiclass task, our method performs significantly lower than batch train. This is not too surprising, for two reasons: i) Training a CNN architecture takes many more epochs over the data to converge (∼ 20 in our case) which is not achieved in a bandit setting; ii) CIFAR-10 has 10 classes, each defining an arm and in our setting; the bandit algorithms only gets feedback for one class in each round, compared to the full feedback received in batch train. Effectively, the number of labels per class in cut by a factor of 10. This is not as much an issue in the pool task, where we can see the between batch train and the bandit algorithms are comparable. In this paper we proposed a new method for training neural networks in a bandit setting. We tackle the problem of exploration-exploitation by estimating uncertainty only in the last layer, allowing the method to scale to large state-of-the-art models. We take advantage of having a posterior over the last layer weights by optimizing the rest of the network over all values of the last layer. We show that method outperforms other methods across a diverse set of underlying models, especially in online tasks where the distribution shifts rapidly. We leave it as future work to investigate more sophisticated methods for determining when to retrain the network, how to set the weight (β) of the regularization term in a more automatic way, and its possible connections to methods used for continual learning. We utilize the same notation here as in Section 3. First, we show how to arrive at equation from the main objective, equation FORMULA1. DISPLAYFORM0 By marginalizing over our zero mean Gaussian prior and using equation FORMULA2, it follows that the last expectation is approximately a constant, and can be removed. The second expectation is equal to the negative cross entropy between Q = p(z|D t, D t−1, θ t−1) and P θ = p(z|D t−1, θ). Using the fact that KL(Q||P θ) = CE(Q||P θ) − H(Q), we can replace the negative cross entropy with −KL(Q||P θ) − H(Q), where H(Q) is the entropy of Q which is constant and can be ignored, yielding equation.We next derive the KL term. As mentioned, we approximate the KL term locally around θ t−1 with a second order Taylor expansion. Again, let K(θ) = KL(p(z|D t, D t−1, θ t−1)||p(z|D t−1, θ)). The second order Taylor expansion around θ = θ t−1 is: DISPLAYFORM1 We can rewrite K (θ) with respect to θ as: DISPLAYFORM2 This can also be done for K. Let ∇ i indicate either the gradient (i = 1) or the Hessian (i = 2). Then we can rewrite the above expectation with respect to the distribution P = p(z|D t−1, θ t−1) instead: DISPLAYFORM3 Using the fact that p(z|D t, D t−1, θ t−1) = p(D t |θ t−1) −1 p(D t |z, θ t−1)p(z|D t−1, θ t−1), we can rewrite the above as: DISPLAYFORM4 the UCB is straightforward; if Σ is the covariance of our posterior over z, and x is the context, then with probability at least 1 − δ, the term (1 + ln(2/δ)/2) f θ (x) T Σf θ (x) is an upper confidence bound. The term α = (1 + ln(2/δ)/2) is treated as a hyperparameter. The arm c chosen is the one whose parameter vector z c maximizes the following: DISPLAYFORM5 The hyperparameter values that are used for all methods across all tasks (the global hyperparameters) are presented in TAB6. The hyper parameters for the low dimensional bandit tasks (we uses the same values for each low dimensional dataset), the paraphrase bandit (multiclass and pool), and the image classification bandit (multiclass and pool) are presented in Table 5. The meanings of the non obvious hyperparameter values are described below:Retrain Epochs: For Sample New and Marginalize; how many times to pass over the new data D t.Update Frequency: How many online rounds to do before updating the rest of the network offline. Num Updates: For Sample All; the number of batches to uniformly sample (and update with) from the entire distribution. Table 5: Method specific experiment details for all tasks. In this section we detail the architectures used for each task. We use the same underlying model for each method. Low dimensional task models. For the Low dimensional tasks, we utilize the same exact Multi Layer Perceptron (MLP) models as used in BID17. The model is a 2 layer MLP with a hidden layer dimension of 100, with ReLu activation functions. Paraphrase task. The paraphrase task uses the same BiMPM architecture and parameters used in. We refer readers to the corresponding paper for details. Image classification task. The convolutions architecture we use is defined as: (i) Two 3 × 3 convolutional layers with 64 filters and ReLu activations; (ii) A 2 × 2 max pooling layer with a stride of 2; (iii) Dropout layer with drop probability 0.25; (iv) A 3 × 3 and 2 × 2 convolutional layer both with 128 filters and ReLu activations; (v) A 2 × 2 max pooling layer with a stride of 2; (vi) Dropout layer with drop probability 0.25; (vii) A fully connected layer with 1024 units, followed by a tanh activation, followed by another fully connected layer with 100 units and a tanh activation. We utilize tanh activations at the end as per , who note that ReLu activations lead to difficulties in estimating the uncertainty.

Represent the short summary for finding the corresponding scientific article (long)

This paper proposes a new method for neural network learning in online bandit settings by marginalizing over the last layer

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.