Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales. Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments. Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (~0.1 ms to ~100 s), a process we call Wave2Midi2Wave. This large advance in the state of the art is enabled by our release of the new MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) dataset, composed of over 172 hours of virtuosic piano performances captured with fine alignment (~3 ms) between note labels and audio waveforms. The networks and the dataset together present a promising approach toward creating new expressive and interpretable neural models of music. Since the beginning of the recent wave of deep learning research, there have been many attempts to create generative models of expressive musical audio de novo. These models would ideally generate audio that is both musically and sonically realistic to the point of being indistinguishable to a listener from music composed and performed by humans. However, modeling music has proven extremely difficult due to dependencies across the wide range of timescales that give rise to the characteristics of pitch and timbre (short-term) as well as those of rhythm (medium-term) and song structure (long-term). On the other hand, much of music has a large hierarchy of discrete structure embedded in its generative process: a composer creates songs, sections, and notes, and a performer realizes those notes with discrete events on their instrument, creating sound. The division between notes and sound is in many ways analogous to the division between symbolic language and utterances in speech. The WaveNet model by BID18 may be the first breakthrough in generating musical audio directly with a neural network. Using an autoregressive architecture, the authors trained a model on audio from piano performances that could then generate new piano audio sample-bysample. However, as opposed to their highly convincing speech examples, which were conditioned on linguistic features, the authors lacked a conditioning signal for their piano model. The was audio that sounded very realistic at very short time scales (1 or 2 seconds), but that veered off into chaos beyond that. BID4 made great strides towards providing longer term structure to WaveNet synthesis by implicitly modeling the discrete musical structure described above. This was achieved by training a hierarchy of VQ-VAE models at multiple time-scales, ending with a WaveNet decoder to generate piano audio as waveforms. While the are impressive in their ability to capture long-term structure directly from audio waveforms, the ing sound suffers from various artifacts at the fine-scale not present in the unconditional WaveNet, clearly distinguishing it from real musical audio. Also, while the model learns a version of discrete structure from the audio, it is not Transcription: onsets & frames (section 4) Synthesis: conditional WaveNet (section 6) Piano roll (MIDI) Symbolic modelling: transformer (section 5) Event predictionFigure 1: Wave2Midi2Wave system architecture for our suite of piano music models, consisting of (a) a conditional WaveNet model that generates audio from MIDI, (b) a Music Transformer language model that generates piano performance MIDI autoregressively, and (c) a piano transcription model that "encodes" piano performance audio as MIDI.directly reflective of the underlying generative process and thus not interpretable or manipulable by a musician or user. BID9 propose a model that uses a WaveNet to generate solo cello music conditioned on MIDI notation. This overcomes the inability to manipulate the generated sequence. However, their model requires a large training corpus of labeled audio because they do not train a transcription model, and it is limited to monophonic sequences. In this work, we seek to explicitly factorize the problem informed by our prior understanding of the generative process of performer and instrument: DISPLAYFORM0 which can be thought of as a generative model with a discrete latent code of musical notes. Since the latent representation is discrete, and the scale of the problem is too large to jointly train, we split the model into three separately trained modules that are each state-of-the-art in their respective domains:1. Encoder, P (notes|audio): An Onsets and Frames transcription model to produce a symbolic representation (MIDI) from raw audio.2. Prior, P (notes): A self-attention-based music language model BID7 to generate new performances in MIDI format based on those transcribed in.3. Decoder, P (audio|notes): A WaveNet (van den BID18 synthesis model to generate audio of the performances conditioned on MIDI generated in.We call this process Wave2Midi2Wave. One hindrance to training such a stack of models is the lack of large-scale annotated datasets like those that exist for images. We overcome this barrier by curating and publicly releasing alongside this work a piano performance dataset containing well-aligned audio and symbolic performances an order of magnitude larger than the previous benchmarks. In addition to the high quality of the samples our method produces (see https://goo.gl/ magenta/maestro-examples), training a suite of models according to the natural musician/instrument division has a number of other advantages. First, the intermediate representation used is more suitable for human interpretation and manipulation. Similarly, factorizing the model in this way provides better modularity: it is easy to independently swap out different performance and instrument models. Using an explicit performance representation with modern language models also allows us to model structure at much larger time scales, up to a minute or so of music. Finally, we can take advantage of the large amount of prior work in the areas of symbolic music generation and conditional audio generation. And by using a state-of-the-art music transcription model, we can make use of the same wealth of unlabeled audio recordings previously only usable for training end-to-end models by transcribing unlabeled audio recordings and feeding them into the rest of our model. Our contributions are as follows:1. We combine a transcription model, a language model, and a MIDI-conditioned WaveNet model to produce a factorized approach to musical audio modeling capable of generating about one minute of coherent piano music.2. We provide a new dataset of piano performance recordings and aligned MIDI, an order of magnitude larger than previous datasets.3. Using an existing transcription model architecture trained on our new dataset, we achieve state-of-the-art on a piano transcription benchmark. We partnered with organizers of the International Piano-e-Competition 1 for the raw data used in this dataset. During each installment of the competition, virtuoso pianists perform on Yamaha Disklaviers which, in addition to being concert-quality acoustic grand pianos, utilize an integrated high-precision MIDI capture and playback system. Recorded MIDI data is of sufficient fidelity to allow the audition stage of the competition to be judged remotely by listening to contestant performances reproduced over the wire on another Disklavier instrument. The dataset introduced in this paper, which we name MAESTRO ("MIDI and Audio Edited for Synchronous TRacks and Organization"), contains over a week of paired audio and MIDI recordings from nine years of International Piano-e-Competition events.2 The MIDI data includes key strike velocities and sustain pedal positions. Audio and MIDI files are aligned with ≈3 ms accuracy and sliced to individual musical pieces, which are annotated with composer, title, and year of performance. Uncompressed audio is of CD quality or higher (44.1-48 kHz 16-bit PCM stereo). A train/validation/test split configuration is also proposed, so that the same composition, even if performed by multiple contestants, does not appear in multiple subsets. Repertoire is mostly classical, including composers from the 17 th to early 20 th century. MusicNet BID16 contains recordings of human performances, but separatelysourced scores. As discussed in, the alignment between audio and score is not fully accurate. One advantage of MusicNet is that it contains instruments other than piano (not counted in table 2) and a wider variety of recording environments. MAPS BID5 contains Disklavier recordings and synthesized audio created from MIDI files that were originally entered via sequencer. As such, the "performances" are not as natural as the MAESTRO performances captured from live performances. In addition, synthesized audio makes up a large fraction of the MAPS dataset. MAPS also contains syntheses and recordings of individual notes and chords, not counted in Our goal in processing the data from International Piano-e-Competition was to produce pairs of audio and MIDI files time-aligned to represent the same musical events. The data we received from the organizers was a combination of MIDI files recorded by Disklaviers themselves and WAV audio captured with conventional recording equipment. However, because the recording streams were independent, they differed widely in start times and durations, and they were also subject to jitter. Due to the large volume of content, we developed an automated process for aligning, slicing, and time-warping provided audio and MIDI to ensure a precise match between the two. Our approach is based on globally minimizing the distance between CQT frames from the real audio and synthesized MIDI (using FluidSynth 3). Obtaining a highly accurate alignment is non-trivial, and we provide full details in the appendix. For all experiments in this paper, we use a single train/validation/test split designed to satisfy the following criteria:• No composition should appear in more than one split.• Train/validation/test should make up roughly 80/10/10 percent of the dataset (in time), respectively. These proportions should be true globally and also within each composer. Maintaining these proportions is not always possible because some composers have too few compositions in the dataset.• The validation and test splits should contain a variety of compositions. Extremely popular compositions performed by many performers should be placed in the training split. For comparison with our , we recommend using the splits which we have provided. We do not necessarily expect these splits to be suitable for all purposes; future researchers are free to use alternate experimental methodologies. The large MAESTRO dataset enables training an automatic piano music transcription model that achieves a new state of the art. We base our model on Onsets and Frames, with several modifications informed by a coarse hyperparameter search using the validation split. For full details of the base model architecture and training procedure, refer to.One important modification was adding an offset detection head, inspired by BID8. The offset head feeds into the frame detector but is not directly used during decoding. The offset labels are defined to be the 32ms following the end of each note. We also increased the size of the bidirectional LSTM layers from 128 to 256 units, changed the number of filters in the convolutional layers from 32/32/64 to 48/48/96, and increased the units in the fully connected layer from 512 to 768. We also stopped gradient propagation into the onset subnetwork from the frame network, disabled weighted frame loss, and switched to HTK frequency spacing BID21 for the mel-frequency spectrogram input. In general, we found that the best ways to get higher performance with the larger dataset were to make the model larger and simpler. The final important change we made was to start using audio augmentation during training using an approach similar to the one described in BID10. During training, every input sample was modified using random parameters for the SoX 4 audio tool using pysox BID2. The parameters, ranges, and random sampling methods are described in table 3. We found that this was particularly important when evaluating on the MAPS dataset, likely because the audio augmentation made the model more robust to differences in recording environment and piano qualities. The differences in training are summarized in Table 4: Transcription Precision, Recall, and F1 Results on MAPS configuration 2 test dataset (ENSTDkCl and ENSTDkAm full-length .wav files). Training was done on the MAESTRO trianing set with audio augmentation. Scores are calculated using the same method as in. Note-based scores calculated by the mir eval library, frame-based scores as defined in BID1. Final metric is the mean of scores calculated per piece. In sections 5 and 6, we demonstrate how using this transcription model enables training language and synthesis models on a large set of unlabeled piano data. To do this, we transcribe the audio in the MAESTRO training set, although in theory any large set of unlabeled piano music would work. We For our generative language model, we use the decoder portion of a Transformer BID20 with relative self-attention, which has previously shown compelling in generating music with longer-term coherence BID7. We trained two models, one on MIDI data from the MAESTRO dataset and another on MIDI transcriptions inferred by Onsets and Frames from audio in MAESTRO, referred to as MAESTRO-T in section 4. For full details of the model architecture and training procedure, refer to BID7.We used the same training procedure for both datasets. We trained on random crops of 2048 events and employed transposition and time compression/stretching data augmentation. The transpositions were uniformly sampled in the range of a minor third below and above the original piece. The time stretches were at discrete amounts and uniformly sampled from the set {0.95, 0.975, 1.0, 1.025, 1.05}.We evaluated both of the models on their respective validation splits. Model variation NLL on their respective validation splits Music Transformer trained on MAESTRO 1.84 Music Transformer trained on MAESTRO-T 1.72 Table 7: Validation Negative Log-Likelihood, with event-based representation. Samples outputs from the Music Transformer model can be heard in the Online Supplement (https://goo.gl/magenta/maestro-examples). Most commercially available systems that are able to synthesize a MIDI sequence into a piano audio signal are concatenative: they stitch together snippets of audio from a large library of recordings of individual notes. While this stitching process can be quite ingenious, it does not optimally capture the various interactions between notes, whether they are played simultaneously or in sequence. An alternative but less popular strategy is to simulate a physical model of the instrument. Constructing an accurate model constitutes a considerable engineering effort and is a field of research by itself BID0 BID17.WaveNet (van den) is able to synthesize realistic instrument sounds directly in the waveform domain, but it is not as adept at capturing musical structure at timescales of seconds or longer. However, if we provide a MIDI sequence to a WaveNet model as conditioning information, we eliminate the need for capturing large scale structure, and the model can focus on local structure instead, i.e., instrument timbre and local interactions between notes. Conditional WaveNets are also used for text-to-speech (TTS), and have been shown to excel at generating realistic speech signals conditioned on linguistic features extracted from textual data. This indicates that the same setup could work well for music audio synthesis from MIDI sequences. Our WaveNet model uses a similar autoregressive architecture to van den Oord et al. FORMULA0, but with a larger receptive field: 6 (instead of 3) sequential stacks with 10 residual block layers each. We found that a deeper context stack, namely 2 stacks with 6 layers each arranged in a series, worked better for this task. We also updated the model to produce 16-bit output using a mixture of logistics as described in van den.The input to the context stack is an onset "piano roll" representation, a size-88 vector signaling the onset of any keys on the keyboard, with 4ms bins (250Hz). Each element of the vector is a float that represents the strike velocity of a piano key in the 4ms frame, scaled to the range. When there is no onset for a key at a given time, the value is 0.We initially trained three models:Unconditioned Trained only with the audio from the combined MAESTRO training/validation splits with no conditioning signal. Ground Trained with the ground truth audio/MIDI pairs from the combined MAESTRO training/validation splits. Transcribed Trained with ground truth audio and MIDI inferred from the audio using the Onsets and Frames method, referred to as MAESTRO-T in section 4.The ing losses after 1M training steps were 3.72, 3.70 and 3.84, respectively. Due to teacher forcing, these numbers do not reflect the quality of conditioning, so we rely on human judgment for evaluation, which we address in the following section. It is interesting to note that the WaveNet model recreates non-piano subtleties of the recording, including the response of the room, breathing of the player, and shuffling of listeners in their seats. These are encouraging and indicate that such methods could also capture the sound of more dynamic instruments (such as string and wind instruments) for which convincing synthesis/sampling methods lag behind piano. Due to the heterogeneity of the ground truth audio quality in terms of microphone placement, ambient noise, etc., we sometime notice "timbral shifts" during longer outputs from these models. We therefore additionally trained a model conditioned on a one-hot year vector at each timestep (similar to speaker conditioning in TTS), which succeeds in producing consistent timbres and ambient qualities during long outputs (see Online Supplement).A side effect of arbitrary windowing of the training data across note boundaries is a sonic crash that often occurs at the beginning of generated outputs. To sidestep this issue, we simply trim the first 2 seconds of all model outputs reported in this paper, and in the Online Supplement (https: //goo.gl/magenta/maestro-examples). Since our ultimate goal is to create realistic musical audio, we carried out a listening study to determine the perceived quality of our method. To separately assess the effects of transcription, language modeling, and synthesis on the listeners' responses, we presented users with two 20-second clips WaveNet Unconditioned Clips generated by the Unconditioned WaveNet model described in section 6. WaveNet Ground/Test Clips generated by the Ground WaveNet model described in section 6, conditioned on random 20-second MIDI subsequences from the MAESTRO test split. WaveNet Transcribed/Test Clips generated by the Transcribed WaveNet model described in section 6, conditioned on random 20-second subsequences from the MAESTRO test split. WaveNet Transcribed/Transformer Clips generated by the Transcribed WaveNet model described in section 6, conditioned on random 20-second subsequences from the Music Transformer model described in section 5 that was trained on MAESTRO-T.The final set of samples demonstrates the full end-to-end ability of taking unlabeled piano performances, inferring MIDI labels via transcription, generating new performances with a language model trained on the inferred MIDI, and rendering new audio as though it were played on a similar piano-all without any information other than raw audio recordings of piano performances. Participants were asked which clip they thought sounded more like a recording of somebody playing a musical piece on a real piano, on a Likert scale. 640 ratings were collected, with each source involved in 128 pair-wise comparisons. FIG0 shows the number of comparisons in which performances from each source were selected as more realistic. A Kruskal-Wallis H test of the ratings showed that there is at least one statistically significant difference between the models: χ 2 = 67.63, p < 0.001. A post-hoc analysis using the Wilcoxon signed-rank test with Bonferroni correction showed that there was not a statistically significant difference in participant ratings between real recordings and samples from the WaveNet Ground/Test and WaveNet Transcribed/Test models with p > 0.01/10.Audio of some of the examples used in the listening tests is available in the Online Supplement (https://goo.gl/magenta/maestro-examples). We have demonstrated the Wave2Midi2Wave system of models for factorized piano music modeling, all enabled by the new MAESTRO dataset. In this paper we have demonstrated all capabilities on the same dataset, but thanks to the new state-of-the-art piano transcription capabilities, any large set of piano recordings could be used, 6 which we plan to do in future work. After transcribing the recordings, the transcriptions could be used to train a WaveNet and a Music Transformer model, and then new compositions could be generated with the Transformer and rendered with the WaveNet. These new compositions would have similar musical characteristics to the music in the original dataset, and the audio renderings would have similar acoustical characteristics to the source piano. The most promising future work would be to extend this approach to other instruments or even multiple simultaneous instruments. Finding a suitable training dataset and achieving sufficient transcription performance will likely be the limiting factors. The new dataset (MIDI, audio, metadata, and train/validation/test split configurations) is available at https://g.co/magenta/maestro-datasetunder a Creative Commons Attribution NonCommercial Share-Alike 4.0 license. The Online Supplement, including audio examples, is available at https://goo.gl/magenta/maestro-examples. We would like to thank Michael E. Jones and Stella Sick for their help in coordinating the release of the source data and Colin Raffel for his careful review and comments on this paper. In this appendix, we describe in detail how the MAESTRO dataset from section 3 was aligned and segmented. The key idea for the alignment process was that even an untrained human can recognize whether two performances are of the same score based on raw audio, disregarding differences in the instrument or recording equipment used. Hence, we synthesized the provided MIDI (using FluidSynth with a SoundFont sampled from recordings of a Disklavier 7) and sought to define an audio-based difference metric that could be minimized to find the best-alignment shift for every audio/MIDI pair. We wanted the metric to take harmonic features into account, so as a first step we used librosa BID10 to compute the Constant-Q Transform BID3 BID15 of both original and synthesized audio. For the initial alignment stage we picked a hop length of 4096 samples (∼90 ms) as a trade-off between speed and accuracy, which proved reasonable for most of the repertoire.8 Microphone setup varied between competition years and stages, ing in varying frequency response and overall amplitude levels in recordings, especially in the lower and higher ends of the piano range. To account for that, we limited the CQT to 48 buckets aligned with MIDI notes C2-B5, and also converted amplitude levels to dB scale with maximum absolute amplitude as a reference point and a hard cut-off at -80 dB. Original and synthesized audio also differed in sound decay rate, so we normalized the ing CQT arrays time-wise by dividing each hop column by its minimum value (averaged over a 5-hop window).A single MIDI file from a Disklavier typically covered several hours of material corresponding to a sequence of shorter audio files from several seconds up to an hour long. We slid the normalized CQT of each such original audio file against a window of synthesized MIDI CQT of the same length and used mean squared error (MSE) between the two as the difference metric.9 Minimum error determined best alignment, after which we attempted to align the next audio file in sequence with the remaining length of the corresponding MIDI file. Due to the length of MIDI files, it was impractical to calculate MSE at each possible shift, so instead we trimmed silence at the beginning of audio, and attempted to align it with the first "note on" event of the MIDI file, within ±12 minutes tolerance. If the minimum error was still high, we attempted alignment at the next "note on" event after a 30-second silence. This approach allowed us to skip over unusable sections of MIDI recordings that did not correspond to audio, e.g., instrument tuning and warm-ups, and also non-musical segments of audio such as applause and announcements. Non-piano sounds also considerably increased the MSE metric for very short audio files, so we had to either concatenate those with their longer neighbors if they had any musical material or exclude them completely. Events that were present at the beginning of audio files beyond the chosen shift tolerance which did not correspond to MIDI had to be cut off manually. In order to recover all musically useful data we also had to manually repair several MIDI files where the clock had erroneously jumped, causing the remainder of the file to be out of sync with corresponding audio. After tuning process parameters and addressing the misaligned audio/MIDI pairs detected by unusually high CQT MSE, we have reached the state where each competition year (i.e., different audio recording setup) has final metric values for all pairs within a close range. Spot-checking the pairs with the highest MSE values for each year confirmed proper alignment, which allowed us to proceed to the segmentation stage. Since certain compositions were performed by multiple contestants, 10 we needed to segment the aligned audio/MIDI pairs further into individual musical pieces, so as to enable splitting the data into train, validation, and test sets disjoint on compositions. While the organizers provided the list of composition metadata for each audio file, for some competition years timing information was missing. In such cases we greedily sliced audio/MIDI pairs at the longest silences between MIDI notes up to the expected number of musical pieces. Where expected piece duration data was available, we applied search with backtracking roughly as follows. As an invariant, the segmentation algorithm maintained a list of intervals as start-end time offsets along with a list of expected piece durations, so that the total length of the piece durations corresponding to each interval was less than the interval duration (within a certain tolerance). At each step we picked the next longest MIDI silence and determined which interval it belonged to. Then we split that interval in two at the silence and attempted to split the corresponding sequence of durations as well, satisfying the invariant. For each suitable split the algorithm continued to the next longest silence. If multiple splits were possible, the algorithm preferred the ones that divided the piece durations more evenly according to a heuristic. If no such split was possible, the algorithm either skipped current silence if it was short 11 and attempted to split at the next one, or backtracked otherwise. It also backtracked if no more silences longer than 3 seconds were available. The algorithm succeeded as soon as each interval corresponded to exactly one expected piece duration. Once a suitable segmentation was found, we sliced each audio/MIDI pair at ing intervals, additionally trimming short clusters of notes at the beginning or end of each segment that appeared next to long MIDI silences in order to cut off additional non-music events (e.g., tuning or contestants testing the instrument during applause), and adding an extra 1 second of padding at both ends before making the final cut. After the initial alignment and segmentation, we applied Dynamic Time Warping (DTW) to account for any jitter in either the audio or MIDI recordings. DTW has seen wide use in audio-to-MIDI alignment; for an overview see BID11. We follow the align midi example from pretty midi BID13, except that we use a custom C++ DTW implementation for improved speed and memory efficiency to allow for aligning long sequences. First, in Python, we use librosa to load the audio and resample it to a 22,050Hz mono signal. Next, we load the MIDI and synthesize it at the same sample rate, using the same FluidSynth process as above. Then, we pad the end of the shorter of the two sample arrays so they are the same length. We use the same procedure as align midi to extract CQTs from both sample arrays, except that we use a hop length of 64 to achieve a resolution of ∼3ms. We then pass these CQTs to our C++ DTW implementation. To avoid calculating the full distance matrix and taking its mean to get a penalty value, we instead sample 100k random pairs and use the mean of their cosine distances. We use the same DTW algorithm as implemented in librosa except that we calculate cosine distances only within a Sakoe-Chiba band radius BID14 of 2.5 seconds instead of calculating distances for all pairs. Staying within this small band limits the number of calculations we need to make and the number of distances we have to store in memory. This is possible because we know from the previous alignment pass that the sequences are already mostly aligned and we just need to account for small constant offsets due to the lower resolution of the previous process and apply small sequence warps to recover from any occasional jitter.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

We train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure, enabled by the new MAESTRO dataset.

Represent the short summary for finding the corresponding scientific article (long)

A way to generate training corpora for neural code synthesis using a discriminator trained on unlabelled data


Question: Represent the scientific article to find a short TLDR

Question: Variational inference based on chi-square divergence minimization (CHIVI) provides a way to approximate a model's posterior while obtaining an upper bound on the marginal likelihood. However, in practice CHIVI relies on Monte Carlo (MC) estimates of an upper bound objective that at modest sample sizes are not guaranteed to be true bounds on the marginal likelihood. This paper provides an empirical study of CHIVI performance on a series of synthetic inference tasks. We show that CHIVI is far more sensitive to initialization than classic VI based on KL minimization, often needs a very large number of samples (over a million), and may not be a reliable upper bound. We also suggest possible ways to detect and alleviate some of these pathologies, including diagnostic bounds and initialization strategies. Estimating the marginal likelihood in probabilistic models is the holy grail of Bayesian inference. Marginal likelihoods allow us to compute the posterior probability of model parameters or perform Bayesian model selection . While exact computation of the marginal is not tractable for most models, variational inference (VI) offers a promising and scalable approximation. VI suggests choosing a simple family of approximate distributions q and then optimizing the parameters of q to minimize its divergence from the true (intractable) posterior. The canonical choice is the KL divergence, where minimizing corresponds to tightening a lower bound on the marginal likelihood. Recently, (a) showed that minimizing a χ 2 divergence leads to a chi-divergence upper bound ("CUBO"). Practitioners often wish to combine upper and lower bound estimates to "sandwich" the model evidence in a narrow range for later decision making, so the CUBO's flexible applicability to all latent variable models is appealing. However, both the estimation of the upper bound and computing its gradient for minimization require Monte Carlo estimators to approximate tough integrals. These estimators may have large variance even at modest number of samples. A natural question is then how reliable CUBO minimization is in practice. In this paper, we provide empirical evidence that CUBO optimization is often tricky, and the bound itself ends up being too loose even Figure 1: Minimizing χ 2 divergence using MC gradient estimates via the reparametrization trick can be challenging even with simple univariate Gaussian distributions. Each column shows under a different number of MC samples. The last column compares ELBO and CUBO traces for S = 10 4; diamonds correspond to sanity-check estimator from Eq.. Top row: variational parameter traces with fixed true variance but changing starting mean locations. Bottom row: same, but with fixed true mean and changing start variance values. using hundreds of samples. Our contributions include: i) evaluation of the CUBO in two simple scenarios, and comparison to other bounds to gauge its utility; ii) empirical analysis of CUBO optimization in both scenarios, in terms of convergence rate and sensitivity to the number of samples; iii) review of alternative upper bounds and best practices for diagnosing and testing new bounds. Let p(x, z) be the joint distribution of observed variables x and latent variables z. Variational inference (VI) approximates the posterior distribution p(z|x) through optimization. The idea is to posit a family of variational distributions and find the member distribution q(z; λ) which is as close as possible to the true posterior. Standard VI minimizes the KL divergence D KL q(z; λ)||p(z|x). Minimizing the KL divergence is equivalent to maximizing the evidence lower bound (ELBO) on the model evidence log p(x). Alternatively, χ 2 variational inference (b) minimizes the χ 2 divergence D KL p(z|x)||q(z; λ). This is equivalent to minimizing the following upper bound (CUBO): The expectation in the CUBO is usually intractable, so we use Monte Carlo samples to construct a biased estimate where z,..., z (S) ∼ q(z; λ). In this paper, we consider two optimization strategies, both relying on the reparametrization trick (; ; Titsias and Lázaro-): i) optimizing the CUBO directly in Eq. using biased gradient estimators; ii) optimizing the exponentiated CUBO defined as L EXPCUBO (λ) = exp(2 L CUBO (λ)), whose gradients are unbiased but might suffer from higher variance. We consider a simple inference scenario: minimizing the divergence between two univariate Gaussian distributions. We assume no data x, such that the true posterior is just the prior fixed at p(z). = N. We consider two cases: a variational distribution q(z;μ,σ 2) with fixedσ = 1.0 and varying meanμ = {1, 2, 4, 10}, or the other way around, fixedμ = 0.0 and varyingσ = {0.1, 0.5, 2.0, 10.0}. All experiments were performed using stochastic gradient descent and grid-searching the learning rate for each different bound independently in a fine grid between 10 −4 and 1.0. Fig. 1 shows the evolution of the variational parameters over time when minimizing the χ 2 divergence (ChiSq) or maximizing the KL divergence (KL) from different initialization points. While the KL trajectories always converge to the true values, the ChiSq variational parameters fail to converge for 5 out of the 8 cases when the number of MC samples S = 100. If we increase the number of samples S to 1M, 3 out of 8 cases still fail to find the true values. Most alarming, in several cases, e.g., fixed mean and varyingσ initialized at 0.1, the CUBO MC estimator present values below 0 (the true marginal likelihood value), so it is not an upper bound anymore, even with 1M samples. Appendix A show similar pathological behaviors for the exponentiated CUBO case. To assess CUBO correctness, consider an alternative MC estimator that samples from the prior p, rather than from q: where z,..., z (S) ∼ p(z). In general, since CUBO optimization is sensitive to initialization, it is a good practice to do warm initializations, either with MAP estimation or by performing KL optimization first during a few iterations. We consider applying the CUBO training objective to the Latent Dirichlet Allocation (LDA) topic model . We focus on single-document inference, where the length of the document should directly impact posterior uncertainty about which topics are used. We assume that there are K = 3 topics and V = 3 vocabulary words. We are given a set of topic-word probabilities φ where φ kv is the probability of word v under topic k. Each document d is represented by counts of V discrete words or features, x d ∈ Z V +. These counts are generated via a document-specific mixture of K topics, variational inference. In particular, we explore two tasks: (i) estimating upper bounds on the marginal likelihood given a fixed q, and (ii) optimizing q to try to improve such bounds. To assess the reliability of upper bound estimation using approximate distributions, we fit four possible q: one Dirichlet via closed-form updates optimizing the ELBO, and 3 separate Logistic Normal (LN) distributions fit via Monte-Carlo gradient descent steps (see details for each q in the appendix). The 3 LNs are respectively a cold-started optimization of the ELBO, a warm-started optimization of the CUBO, and a cold-started optimization of the CUBO. Warm-starting here means that the mean of q is set to the maximum likelihood estimator of the document-topic vector π d, while cold-starting has random parameters not informed by the data. We hope that these detailed experiments tease apart the impact of initialization and optimization. In Tab. 1 and Tab. 2, for each q described above, we compare CUBO to an alternative upper bound KLpq, detailed in Appendix B. For each stochastic upper bound estimator, we compute 20 replicates using each 100 samples and 100,000 samples, then report the median of these samples as well as 5-th and 95-th percentile value intervals. Our are: CHIVI parameter estimation often diverges for cold initializations. We replicated this issue across many settings, as reported in Tab. 1. CUBO estimators are overconfident. Increasing sample size widens confidence intervals. KLpq estimators are better behaved. Consider Tab. 2's warm-init CUBO row (in Appendix A): At 100 samples the CUBO seems to be within (-1.03, 0.77), but at many more samples, the CUBO interval drops to (-0.86, -0.64), with a new median that is just barely contained in the previous interval. In contrast, the 100 sample KLpq bound has an interval that shrinks. ELBO optimization followed by CUBO computation may be enough. The Dirichlet q optimized for the ELBO but then fitted into a CUBO estimator produces competitive bounds. This suggests that it may not always be necessary to optimize the CUBO directly. Table 2: Topic model case study. Bounds on marginal likelihood for a "short" toy document under an LDA topic model. We infer an approximate posterior over doc-topic probabilities for a single document with just 1 word, using either closed-form coordinate ascent updates to fit a Dirichlet q (top row) or MC gradient updates to fit a LogisticNormal q (bottom rows) with 100 samples per gradient step. Using the final fitted q, we then compute 20 replicates of our stochastic upper bounds on marginal likelihood using either the CUBO or the KLpq estimator (see Appendix B, using S = 10 2 or 10 5 samples for each. We show the median value and the (5%, 95%) interval. Appendix B. The "KLpq" bound: reliable but expensive. Given any approximate posterior q(π d) parameterized byv d ∈ V, the following is an upper bound on the marginal likelihood: show that minimizing this bound is equivalent to minimizing KL(p||q), which computes the asymmetric KL divergence in the opposite direction of typical variational methods, which minimize KL(q||p). We suggest that this bound is a useful comparison point for the CUBO bound. The "KLpq" upper bound can be approximated using S samples from the posterior. For our LDA model, we compute S samples from a Hamiltonian Monte Carlo posterior using Stan . Because LN random variables are not very common, we write the log probability density function of the approximate posterior here, using from Aitchison and Shen (1980, Eq. 1.3 The entropy of the distribution is then: where we have used standard to simplify that last term: This expectation E π d ∼q [log π dk] unfortunately has no closed form. Reparameterization trick. We can write the random variable π d as a deterministic transform of a standard normal random variable u d. First, recall we can map any K − 1-length real vector u ∈ R K−1 to the K-dimensional simplex ∆ K via the softmax transformation: This transformation is one-to-one invertible, and also differentiable w.r.t. its input vector. Now, to generate π d ∈ ∆ K, we can draw π d in three steps: draw u d from a standard normal, scale it with the appropriate mean and standard deviation parameters, and apply the softmax transformation, C.3. LDA Optimization #3: Overcomplete-Logistic-Normal + MonteCarloGD Transformation between overcomplete simplex and the reals We now consider an overcomplete representation of the K-dimensional simplex. Rather than the minimal K − 1 parameters in the LN-Marg approximation above, let's look at transformations that use K free parameters. In this overcomplete space, we must augment our probability vector π d ∈ ∆ K (which has only K − 1 degrees of freedom) with an additional scalar real random variable w d ∈ R, so the combined vector [π d1 . . . π d,K−1 w d] has the required K linearlyindependent dimensions. Now, we can create an invertible transformation between two K-length vectors: a vector u of real values, and the augmented pair π, w: Because this is an invertible transformation, we can compute the Jacobian:...... Next, we wish to compute the determinant of this Jacobian, as a function of π and w. First, we perform row and column swaps until only the first column and first row have non-diagonal entries, like this: Here, we have defined the remaining mass beyond the K − 1 independent entries of the vector π as rem(π) = 1 − K−1 k=1 π k for simplicity. The number of swaps needed to create J from J is always an even number (there will be the some a swaps needed to fix the rows, and then the same number a swaps for the columns, so 2a swaps total). Each single row or column swap changes the sign of the determinant but not the value. An even number of swaps thus leaves the determinant unchanged: |J | = |J|. We can then apply the Schur determinant formula, which says, for any square matrix, we can compute its determinant by manipulating its subcomponent blocks: The simplification arises via algebra after plugging in the definition of rem(π). Armed with the Jacobian and its determinant, we have all the tools needed to perform variational inference in this representation. Approximate posterior: Overcomplete LN. Returning to our topic modeling task, we consider again the LDA generative model for a document as a given, and wish to compute an approximate posterior for the document-topic vector π d. We suggest an approximate posterior family based on the overcomplete logistic normal above. We can draw samples from this in two steps. This leads to the following log probability density function over the joint space of π, w ∈ ∆ K × R: log q(π, w) = log |det J(π, w)| + Our generative model does not include the log-scale variable w d, but we can easily just give it a N prior and keep it decoupled from the data.

Represent the short summary for finding the corresponding scientific article (long)

An empirical study of variational inference based on chi-square divergence minimization, showing that minimizing the CUBO is trickier than maximizing the ELBO

Represent the TLDR for finding the corresponding scientific article

Propose first methods for exactly optimizing the softmax distribution using stochastic gradient with runtime independent on the number of classes or datapoints.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: It has been widely recognized that adversarial examples can be easily crafted to fool deep networks, which mainly root from the locally non-linear behavior nearby input examples. Applying mixup in training provides an effective mechanism to improve generalization performance and model robustness against adversarial perturbations, which introduces the globally linear behavior in-between training examples. However, in previous work, the mixup-trained models only passively defend adversarial attacks in inference by directly classifying the inputs, where the induced global linearity is not well exploited. Namely, since the locality of the adversarial perturbations, it would be more efficient to actively break the locality via the globality of the model predictions. Inspired by simple geometric intuition, we develop an inference principle, named mixup inference (MI), for mixup-trained models. MI mixups the input with other random clean samples, which can shrink and transfer the equivalent perturbation if the input is adversarial. Our experiments on CIFAR-10 and CIFAR-100 demonstrate that MI can further improve the adversarial robustness for the models trained by mixup and its variants. Deep neural networks (DNNs) have achieved state-of-the-art performance on various tasks . However, counter-intuitive adversarial examples generally exist in different domains, including computer vision , natural language processing , reinforcement learning , speech and graph data . As DNNs are being widely deployed, it is imperative to improve model robustness and defend adversarial attacks, especially in safety-critical cases. Previous work shows that adversarial examples mainly root from the locally unstable behavior of classifiers on the data manifolds (; ; 2018; b), where a small adversarial perturbation in the input space can lead to an unreasonable shift in the feature space. On the one hand, many previous methods try to solve this problem in the inference phase, by introducing transformations on the input images. These attempts include performing local linear transformation like adding Gaussian noise , where the processed inputs are kept nearby the original ones, such that the classifiers can maintain high performance on the clean inputs. However, as shown in Fig. 1(a), the equivalent perturbation, i.e., the crafted adversarial perturbation, is still δ and this strategy is easy to be adaptively evaded since the randomness of x 0 w.r.t x 0 is local . Another category of these attempts is to apply various non-linear transformations, e.g., different operations of image processing . They are usually off-the-shelf for different classifiers, and generally aim to disturb the adversarial perturbations, as shown in Fig. 1(b). Yet these methods are not quite reliable since there is no illustration or guarantee on to what extent they can work. On the other hand, many efforts have been devoted to improving adversarial robustness in the training phase. For examples, the adversarial training (AT) methods induce locally stable behavior via data augmentation on adversarial examples. However, AT methods are usually computationally expensive, and will often degenerate model performance on the clean inputs or under general-purpose transformations like rotation . In contrast, the mixup training method introduces globally linear behavior in-between the data manifolds, which can also improve adversarial robustness (Zhang Virtual inputs The processed inputs fed into classifiers (a) (b) (c) Figure 1: Intuitive mechanisms in the input space of different input-processing based defenses. x is the crafted adversarial example, x0 is the original clean example, which is virtual and unknown for the classifiers. δ is the adversarial perturbation. et al., 2018; a). Although this improvement is usually less significant than it ed by AT methods, mixup-trained models can keep state-of-the-art performance on the clean inputs; meanwhile, the mixup training is computationally more efficient than AT. The interpolated AT method also shows that the mixup mechanism can further benefit the AT methods. However, most of the previous work only focuses on embedding the mixup mechanism in the training phase, while the induced global linearity of the model predictions is not well exploited in the inference phase. Compared to passive defense by directly classifying the inputs, it would be more effective to actively defend adversarial attacks by breaking their locality via the globally linear behavior of the mixup-trained models. In this paper, we develop an inference principle for mixup-trained models, named mixup inference (MI). In each execution, MI performs a global linear transformation on the inputs, which mixups the input x with a sampled clean example x s, i.e.,x = λx + (1 − λ)x s (detailed in Alg. 1), and feedx into the classifier as the processed input. There are two basic mechanisms for robustness improving under the MI operation (detailed in Sec. 3.2.1), which can be illustrated by simple geometric intuition in Fig. 1(c). One is perturbation shrinkage: if the input is adversarial, i.e., x = x 0 + δ, the perturbation δ will shrink by a factor λ after performing MI, which is exactly the mixup ratio of MI according to the similarity between triangles. Another one is input transfer: after the MI operation, the reduced perturbation λδ acts on random x 0. Comparing to the spatially or semantically local randomness introduced by Gaussian noise or image processing,x 0 introduces spatially global and semantically diverse randomness w.r.t x 0. This makes it less effective to perform adaptive attacks against MI . Furthermore, the global linearity of the mixup-trained models ensures that the information of x 0 remained inx 0 is proportional to λ, such that the identity of x 0 can be recovered from the statistics ofx 0. In experiments, we evaluate MI on CIFAR-10 and CIFAR-100 under the oblivious attacks and the adaptive attacks . The demonstrate that our MI method is efficient in defending adversarial attacks in inference, and is also compatible with other variants of mixup, e.g., the interpolated AT method. Note that also propose to mixup the input points in the test phase, but they do not consider their method from the aspect of adversarial robustness. In this section, we first introduce the notations applied in this paper, then we provide the formula of mixup in training. We introduce the adversarial attacks and threat models in Appendix A.1. Given an input-label pair (x, y), a classifier F returns the softmax prediction vector F (x) and the, where L is the number of classes and [L] = {1, · · ·, L}. The classifier F makes a correct prediction on x if y =ŷ. In the adversarial setting, we augment the data pair (x, y) to a triplet (x, y, z) with an extra binary variable z, i.e., The variable z is usually considered as hidden in the inference phase, so an input x (either clean or adversarially corrupted) can be generally denoted as x = x 0 + δ · 1 z=1. Here x 0 is a clean sample from the data manifold p(x) with label y 0, 1 z=1 is the indicator function, and δ is a potential perturbation crafted by adversaries. It is worthy to note that the perturbation δ should not change the true label of the input, i.e., y = y 0. For p -norm adversarial attacks , we have δ p ≤, where is a preset threshold. Based on the assumption that adversarial examples are off the data manifolds, we formally have x 0 + δ / ∈ supp(p(x)) (a). In supervised learning, the most commonly used training mechanism is the empirical risk minimization (ERM) principle , which minimizes with the loss function L. While computationally efficient, ERM could lead to memorization of data and weak adversarial robustness . As an alternative, introduce the mixup training mechanism, which minimizes 1 m m j=1 L(F (x j),ỹ j ). Herex j = λx j0 + (1 − λ)x j1;ỹ j = λy j0 + (1 − λ)y j1, the input-label pairs (x j0, y j0) and (x j1, y j1) are randomly sampled from the training dataset, λ ∼ Beta(α, α) and α is a hyperparameter. Training by mixup will induce globally linear behavior of models in-between data manifolds, which can empirically improve generalization performance and adversarial robustness a; b; a; b). Compared to the adversarial training (AT) methods , trained by mixup requires much less computation and can keep state-of-the-art performance on the clean inputs. Although the mixup mechanism has been widely shown to be effective in different domains (; ; a; b), most of the previous work only focuses on embedding the mixup mechanism in the training phase, while in the inference phase the global linearity of the trained model is not well exploited. Compared to passively defending adversarial examples by directly classifying them, it would be more effective to actively utilize the globality of mixup-trained models in the inference phase to break the locality of adversarial perturbations. The above insight inspires us to propose the mixup inference (MI) method, which is a specialized inference principle for the mixup-trained models. In the following, we apply colored y,ŷ and y s to visually distinguish different notations. Consider an input triplet (x, y, z), where z is unknown in advance. When directly feeding x into the classifier F, we can obtain the predicted labelŷ. In the adversarial setting, we are only interested in the cases where x is correctly classified by F if it is clean, or wrongly classified if it is adversarial . This can be formally denoted as The general mechanism of MI works as follows. Every time we execute MI, we first sample a label y s ∼ p s (y), then we sample x s from p s (x|y s) and mixup it with x asx = λx + (1 − λ)x s. p s (x, y) denotes the sample distribution, which is constrained to be on the data manifold, i.e., supp(p s (x)) ⊂ supp(p(x)). In practice, we execute MI for N times and average the output predictions to obtain F MI (x), as described in Alg. 1. Here we fix the mixup ratio λ in MI as a hyperparameter, while similar properties hold if λ comes from certain distribution. Theoretically, with unlimited capability and sufficient clean samples, a well mixup-trained model F can be denoted as a linear function H on the convex combinations of clean examples , i.e., ∀x i, x j ∼ p(x) and λ ∈, there is Specially, we consider the case where the training objective L is the cross-entropy loss, then H(x i) should predict the one-hot vector of label y i, i.e., H y (Input: The mixup-trained classifier F ; the input x. Hyperparameters: The sample distribution p s ; the mixup ratio λ; the number of execution N . adversarial, then there should be an extra non-linear part G(δ; x 0) of F, since x is off the data manifolds. Thus for any input x, the prediction vector can be compactly denoted as According to Eq. and Eq., the output ofx in MI is given by: where ∞ −→ represents the limitation when the execution times N → ∞. Now we separately investigate the y-th andŷ-th (could be the same one) components of F (x) according to Eq., and see how these two components differ from those of F (x). These two components are critical because they decide whether we can correctly classify or detect adversarial examples . Note that there is H y (x 0) = 1 and H ys (x s) = 1, thus we have the y-th components as Furthermore, according to Eq., there is 1 y=ŷ = 1 z=0. We can represent theŷ-th components as From the above formulas we can find that, except for the hidden variable z, the sampling label y s is another variable which controls the MI output F (x) for each execution. Different distributions of sampling y s in different versions of MI. Here we consider two easy-to-implement cases: MI with predicted label (MI-PL): In this case, the sampling label y s is the same as the predicted labelŷ, i.e., p s (y) = 1 y=ŷ is a Dirac distribution onŷ. In this case, the label y s is uniformly sampled from the labels other thanŷ, i.e., p s (y) = Uŷ(y) is a discrete uniform distribution on the set {y ∈ [L]|y =ŷ}. We list the simplified formulas of Eq. and Eq. under different cases in Table 1 for clear representation. With the above formulas, we can evaluate how the model performance changes with and without MI by focusing on the formula of Specifically, in the general-purpose setting where we aim to correctly classify adversarial examples , we claim that the MI method improves the robustness if the prediction value on the true label y increases while it on the adversarial labelŷ decreases after performing MI when the input is adversarial (z = 1). This can be formally denoted as We refer to this condition in Eq. as robustness improving condition (RIC). Further, in the detection-purpose setting where we want to detect the hidden variable z and filter out adversarial inputs, we can take the gap of theŷ-th component of predictions before and after the MI operation, i.e., ∆Fŷ(x; p s) as the detection metric (a). To formally measure the detection ability on z, we use the detection gap (DG), denoted as A higher value of DG indicates that ∆Fŷ(x; p s) is better as a detection metric. In the following sections, we specifically analyze the properties of different versions of MI according to Table 1, and we will see that the MI methods can be used and benefit in different defense strategies. In the MI-PL case, when the input is clean (i.e., z = 0), there is F (x) = F (x), which means ideally the MI-PL operation does not influence the predictions on the clean inputs. When the input is adversarial (i.e., z = 1), MI-PL can be applied as a general-purpose defense or a detection-purpose defense, as we separately introduce below: General-purpose defense: If MI-PL can improve the general-purpose robustness, it should satisfy RIC in Eq.. By simple derivation and the of Table 1, this means that Since an adversarial perturbation usually suppress the predicted confidence on the true label and promote it on the target label , there should be Gŷ(δ;x 0) > 0 and G y (δ;x 0) < 0. Note that the left part of Eq. can be decomposed into Here Eq. indicates the two basic mechanisms of the MI operations defending adversarial attacks, as shown in Fig. 1(c). The first mechanism is input transfer, i.e., the clean input that the adversarial perturbation acts on transfers from the deterministic x 0 to stochasticx 0. Compared to the Gaussian noise or different image processing methods which introduce spatially or semantically local randomness, the stochasticx 0 induces spatially global and semantically diverse randomness. This will make it harder to perform an adaptive attack in the white-box setting . The second mechanism is perturbation shrinkage, where the original perturbation δ shrinks by a factor λ. This equivalently shrinks the perturbation threshold since λδ p = λ δ p ≤ λ, which means that MI generally imposes a tighter upper bound on the potential attack ability for a crafted perturbation. Besides, empirical in previous work also show that a smaller perturbation threshold largely weakens the effect of attacks . Therefore, if an adversarial attack defended by these two mechanisms leads to a prediction degradation as in Eq., then applying MI-PL would improve the robustness against this adversarial attack. Similar properties also hold for MI-OL as described in Sec. 3.2.2. In Fig. 2, we empirically demonstrate that most of the existing adversarial attacks, e.g., the PGD attack satisfies these properties. Detection-purpose defense: According to Eq., the formula of DG for MI-PL is By comparing Eq. and Eq., we can find that they are consistent with each other, which means that for a given adversarial attack, if MI-PL can better defend it in general-purpose, then ideally MI-PL can also better detect the crafted adversarial examples. As to MI-OL, when the input is clean (z = 0), there would be a degeneration on the optimal clean prediction as F y (x) = Fŷ(x) = λ, since the sampled x s does not come from the true label y. As compensation, MI-OL can better improve robustness compared to MI-PL when the input is adversarial (z = 1), since the sampled x s also does not come from the adversarial labelŷ in this case. General-purpose defense: Note that in the MI-OL formulas of Table 1, there is a term of 1 y=ys. Since we uniformly select y s from the set [L] \ {ŷ}, there is E(1 y=ys) = 1 L−1. According to the RIC, MI-OL can improve robustness against the adversarial attacks if there satisfies Note that the conditions in Eq. is strictly looser than Eq., which means MI-OL can defend broader range of attacks than MI-PL, as verified in Fig. 2. Detection-purpose defense: According to Eq. and Table 1, the DG for MI-OL is It is interesting to note that DG MI-PL = DG MI-OL, thus the two variants of MI have the same theoretical performance in the detection-purpose defenses. However, in practice we find that MI-PL performs better than MI-OL in detection, since empirically mixup-trained models cannot induce ideal global linearity (cf. Fig. 2 in). Besides, according to Eq., to statistically make sure that the clean inputs will be correctly classified after MI-OL, there should be ∀k ∈ [L] \ {y}, In this section, we provide the experimental on CIFAR-10 and CIFAR-100 to demonstrate the effectiveness of our MI methods on defending adversarial attacks. Our code is available at an anonymous link: http://bit.ly/2kpUZVR. In training, we use ResNet-50 and apply the momentum SGD optimizer on both CIFAR-10 and CIFAR-100. We run the training for 200 epochs with the batch size of 64. The initial learning rate is 0.01 for ERM, mixup and AT; 0.1 for interpolated AT. The learning rate decays with a factor of 0.1 at 100 and 150 epochs. The attack method for AT and interpolated AT is untargeted PGD-10 with = 8/255 and step size 2/255 , and the ratio of the clean examples and the adversarial ones in each mini-batch is 1: 1. The hyperparameter α for mixup and interpolated AT is 1.0. All defenses with randomness are executed 30 times to obtain the averaged predictions. To verify and illustrate our theoretical analyses in Sec. 3, we provide the empirical relationship between the output predictions of MI and the hyperparameter λ in Fig. 2. The notations and formulas annotated in Fig. 2 correspond to those introduced in Sec. 3. We can see that the follow our theoretical under the assumption of ideal global linearity. Besides, both MI-PL and MI-OL empirically satisfy RIC in this case, which indicates that they can improve robustness under the untargeted PGD-10 attack on CIFAR-10, as quantitatively demonstrated in the following sections. In this subsection, we evaluate the performance of our method under the oblivious-box attacks . The oblivious threat model assumes that the adversary is not aware of the existence of the defense mechanism, e.g., MI, and generate adversarial examples based on the unsecured classification model. We separately apply the model trained by mixup and interpolated AT as the classification model. The AUC scores for the detection-purpose defense are given in Fig. 3(a). The show that applying MI-PL in inference can better detect adversarial attacks, while directly detecting by the returned confidence without MI-PL performs even worse than a random guess. Table 3: Classification accuracy (%) on the oblivious adversarial examples crafted on 1,000 randomly sampled test points of CIFAR-100. Perturbation = 8/255 with step size 2/255. The subscripts indicate the number of iteration steps when performing attacks. The notation ≤ 1 represents accuracy less than 1%. The parameter settings for each method can be found in Table 5. Methods Cle. PGD10 PGD50 PGD200 PGD10 PGD50 PGD200 We also compare MI with previous general-purpose defenses applied in the inference phase, e.g., adding Gaussian noise or random rotation ; performing random padding or resizing after random cropping (; . The performance of our method and baselines on CIFAR-10 and CIFAR-100 are reported in Table 2 and Table 3, respectively. Since for each defense method, there is a trade-off between the accuracy on clean samples and adversarial samples depending on the hyperparameters, e.g., the standard deviation for Gaussian noise, we carefully select the hyperparameters to ensure both our method and baselines keep a similar performance on clean data for fair comparisons. The hyperparameters used in our method and baselines are reported in Table 4 and Table 5 . In Fig. 3(b), we further explore this trade-off by grid searching the hyperparameter space for each defense to demonstrate the superiority of our method. As shown in these , our MI method can significantly improve the robustness for the trained models with induced global linearity, and is compatible with training-phase defenses like the interpolated AT method. As a practical strategy, we also evaluate a variant of MI, called MI-Combined, which applies MI-OL if the input is detected as adversarial by MI-PL with a default detection threshold; otherwise returns the prediction on the original input. We also perform ablation studies of ERM / AT + MI-OL in Table 2, where no global linearity is induced. The verify that our MI methods indeed exploit the global linearity of the mixup-trained models, rather than simply introduce randomness. , we test our method under the white-box adaptive attacks (detailed in Appendix B.2). Since we mainly adopt the PGD attack framework, which synthesizes adversarial examples iteratively, the adversarial noise will be clipped to make the input image stay within the valid range. It in the fact that with mixup on different training examples, the adversarial perturbation will be clipped differently. To address this issue, we average the generated perturbations over the adaptive samples as the final perturbation. The of the adversarial accuracy w.r.t the number of adaptive samples are shown in Fig. 4. We can see that even under a strong adaptive attack, equipped with MI can still improve the robustness for the classification models. In this section, we provide more s which are related to our work in the main text. Adversarial attacks. Although deep learning methods have achieved substantial success in different domains , human imperceptible adversarial perturbations can be easily crafted to fool high-performance models, e.g., deep neural networks (DNNs) . One of the most commonly studied adversarial attack is the projected gradient descent (PGD) method . Let r be the number of iteration steps, x 0 be the original clean example, then PGD iteratively crafts the adversarial example as where clip x, (·) is the clipping function. Here x * 0 is a randomly perturbed image in the neighborhood of x 0, i.e.,Ů (x 0,), and the finally returned adversarial example is x = x * r = x 0 + δ, following our notations in the main text. Threat models. Here we introduce different threat models in the adversarial setting. As suggested in, a threat model includes a set of assumptions about the adversarys goals, capabilities, and knowledge. Adversary's goals could be simply fooling the classifiers to misclassify, which is referred to as untargeted mode. Alternatively, the goals can be more specific to make the model misclassify certain examples from a source class into a target class, which is referred to as targeted mode. In our experiments, we evaluate under both modes, as shown in Table 2 and Table 3. Adversary's capabilities describe the constraints imposed on the attackers. Adversarial examples require the perturbation δ to be bounded by a small threshold under p -norm, i.e., δ p ≤. For example, in the PGD attack, we consider under the ∞ -norm. Adversary's knowledge describes what knowledge the adversary is assumed to have. Typically, there are three settings when evaluating a defense method: • Oblivious adversaries are not aware of the existence of the defense D and generate adversarial examples based on the unsecured classification model F . • White-box adversaries know the scheme and parameters of D, and can design adaptive methods to attack both the model F and the defense D simultaneously . • Black-box adversaries have no access to the parameters of the defense D or the model F with varying degrees of black-box access. In our experiments, we mainly test under the oblivious setting (Sec. 4.3) and white-box setting (Sec. 4.4), since previous work has already demonstrated that randomness itself is efficient on defending black-box attacks (; . To date, the most widely applied framework for adversarial training (AT) methods is the saddle point framework introduced in: Here θ represents the trainable parameters in the classifier F, and S is a set of allowed perturbations. In implementation, the inner maximization problem for each input-label pair (x, y) is approximately solved by, e.g., the PGD method with different random initialization . As a variant of the AT method, propose the interpolated AT method, which combines AT with mixup. Interpolated AT trains on interpolations of adversarial examples along with interpolations of unperturbed examples (cf. Alg. 1 in . Previous empirical demonstrate that interpolated AT can obtain higher accuracy on the clean inputs compared to the AT method without mixup, while keeping the similar performance of robustness. AT + MI-OL (ablation study) The λOL = 0.8 The λOL = 0.6 We provide more technical details about our method and the implementation of the experiments. Generality. According to Sec. 3, except for the mixup-trained models, the MI method is generally compatible with any trained model with induced global linearity. These models could be trained by other methods, e.g., manifold mixup (a; ; . Besides, to better defend white-box adaptive attacks, the mixup ratio λ in MI could also be sampled from certain distribution to put in additional randomness. Empirical gap. As demonstrated in Fig. 2, there is a gap between the empirical and the theoretical formulas in Table 1 . This is because that the mixup mechanism mainly acts as a regularization in training, which means the induced global linearity may not satisfy the expected behaviors. To improve the performance of MI, a stronger regularization can be imposed, e.g., training with mixup for more epochs, or applying matched λ both in training and inference. , we design the adaptive attacks for our MI method. Specifically, according to Eq., the expected model prediction returned by MI is: Note that generally the λ in MI comes from certain distribution. For simplicity, we fix λ as a hyperparameter in our implementation. Therefore, the gradients of the prediction w.r.t. the input x is: = E ps ∂F (u) ∂u u=λx+(1−λ)xs · ∂λx + (1 − λ)x s ∂x = λE ps ∂F (u) ∂u | u=λx+(1−λ)xs. Table 3. The number of execution for each random method is 30. In the implementation of adaptive PGD attacks, we first sample a series of examples {x s,k} N A k=1, where N A is the number of adaptive samples in Fig. 3. Then according to Eq., the sign of gradients used in adaptive PGD can be approximated by sign ∂F MI (x) ∂x ≈ sign N A k=1 ∂F (u) ∂u u=λx+(1−λ)x s,k. The hyperparameter settings of the experiments shown in Table 2 and Table 3 are provided in Table 4 and Table 5, respectively. Since the original methods in and are both designed for the models on ImageNet, we adapt them for CIFAR-10 and CIFAR-100. Most of our experiments are conducted on the NVIDIA DGX-1 server with eight Tesla P100 GPUs.

We exploit the global linearity of the mixup-trained models in inference to break the locality of the adversarial perturbations.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the scientific article to find a short TLDR

Question: Fine-tuning language models, such as BERT, on domain specific corpora has proven to be valuable in domains like scientific papers and biomedical text. In this paper, we show that fine-tuning BERT on legal documents similarly provides valuable improvements on NLP tasks in the legal domain. Demonstrating this outcome is significant for analyzing commercial agreements, because obtaining large legal corpora is challenging due to their confidential nature. As such, we show that having access to large legal corpora is a competitive advantage for commercial applications, and academic research on analyzing contracts. Businesses rely on contracts to capture critical obligations with other parties, such as: scope of work, amounts owed, and cancellation policies. Various efforts have gone into automatically extracting and classifying these terms. These efforts have usually been modeled as: classification, entity and relation extraction tasks. In this paper we focus on classification, but in our application we have found that our findings apply equally and sometimes, more profoundly, on other tasks. Recently, numerous studies have shown the value of fine-tuning language models such as ELMo and BERT to achieve state-of-the-art on domain specific tasks. In this paper we investigate and quantify the impact of utilizing a large domain-specific corpus of legal agreements to improve the accuracy of classification models by fine-tuning BERT. Specifically, we assess: (i) the performance of a simple model that only uses the pre-trained BERT language model, (ii) the impact of further fine tuning BERT, and (iii) how this impact changes as we train on larger corpora. Ultimately, our investigations show marginal, but valuable, improvements that increase as we grow the size of the legal corpus used to fine-tine BERT -and allow us to confidently claim that not only is this approach valuable for increasing accuracy, but commercial enterprises seeking to create these models will have an edge if they can amass a corpus of legal documents. Lexion is commercial venture that is building an "intelligent repository" for legal agreements that automatically classifies documents and then, based on the document type, fills a schema of metadata values using entity extraction, classification, and relationship extraction. Our application then uses this metadata to perform a variety of tasks that are valuable to end users: automatically organizing documents; linking related documents; calculating date milestones; identifying outlier terms; and a host of features to run reports, receive alerts, share with permissions, and integrate with other systems. (See Fig 1, screenshot). To deliver this application, we have developed an extensive pipeline and user-interface to ingest raw documents, perform OCR with multiple error detection and cleanup steps, rapidly annotate thousands of documents in hours, and train and deploy several models. Delivering the most accurate models possible, while managing our annotation costs, is an important challenge for us. Furthermore, we wish to leverage the massive legal corpus that we have acquired, and turn it into a competitive advantage using unsupervised techniques. For these reasons, applying pre-trained language models, and fine-tuning them further on our legal corpus, is an attractive approach to maximize accuracy and provide a more beneficial solution than our competitors. To fine-tune BERT, we used a proprietary corpus that consists of hundreds of thousands of legal agreements. We extracted text from the agreements, tokenized it into sentences, and removed sentences without alphanumeric text. We selected the BERT-Base uncased pre-trained model for fine-tuning. To avoid including repetitive content found at the beginning of each agreement we selected the 31st to 50th sentence of each agreement. We ran unsupervised fine-tuning of BERT using sequence lengths of 128, 256 and 512. The loss function over epochs is shown in Figure 2. We used a proprietary dataset consisting of a few thousand legal agreements. These were hand annotated by our model-development team using our internal rapid-annotation tools. We annotate a few dozen attributes per document, but for this paper we hand picked a single common and high value class: the "Term" of an agreement. In practice, the term of agreement can be one of about half a dozen possible classes, but we chose to focus on the two most common classes for this research: the "fixed" term, i.e. the term of an agreement that expires after a fixed amount of time; and the "auto-renewing" term, i.e. the term of an agreement that automatically renews. While this attribute might seem simple at a glance, there are many subtleties that make it challenging to extract this with a high enough accuracy for practical applications. Our end-to-end system does a great deal of preand post-processing to achieve an impressive level of accuracy that makes our application viable for end users, the details of which are beyond the scope of this paper. We split our classification dataset into train (80%) and validation (20%) sets. For all architecture variations, we train for a variable number of epochs as long as the validation error is decreasing. We stop training when validation error starts increasing again and then report the final on a held-out test set. In doing so we try to avoid over-fitting on the training set. For a baseline, we trained a simple neural network with the architecture shown in figure 5. The input to the network was a Bag-of-Word representation of the text. The BERT classifier we used consisted of the BERT layers, followed by the last three layers of our baseline network shown in figure 4. When training our BERT-based models, we also fine tuned the BERT layers on the end task. In order to assess the delta from using the Language Model (LM) that was fine-tuned on our legal corpus, we performed another experiment where we froze the BERT layers and only trained the last portion of the network. While the final accuracy of this model was sub par, even compared to our baseline model, the gains from using a fine-tuned instead of a pre-trained LM is much more pronounced, providing further evidence for the value of domain-specific fine tuning. These are shown in Table 1. We use 4 metrics to compare performance across various experiments: Matthews Correlation Coefficient, as well as Precision, Recall and F1 score weighted by class size. In table 2 we show the various we got from different configurations. It's clear that using pre-trained BERT, we're able to achieve a significant performance lift compared to the base line. It is also clear that fine-tuning BERT on a domain-specific corpus noticeably improves this lift, even when the corpus size is small and we train for a short time. In Figure 3 we also show the different rates of change in train loss across epochs between pre-trained BERT vs fine-tuned Bert. As shown, the model trained on the fine-tuned version learns faster as evident in the faster drop in train loss on the training set (note the logarithmic y-axis). It is worth mentioning that our BERT-based architecture is very simplistic for the sake of a fair comparison. In practice, having a deeper neural network on top of BERT that is specialized in the end task yields much more impressive , and that's the architecture we use in our system. We show the of using a slightly more advanced architecture with fine-tuned BERT in table 3 to demonstrate what's possible without any sophisticated feature engineering or hyper-parameter tuning.

Fine-tuning BERT on legal corpora provides marginal, but valuable, improvements on NLP tasks in the legal domain.

Represent the short summary for finding the corresponding scientific article (long):

We can identify prototypical and outlier examples in machine learning that are quantifiably very different, and make use of them to improve many aspects of neural networks.