Question: Represent the article for finding a single-sentence TLDR

Question: Recent studies have highlighted adversarial examples as a ubiquitous threat to different neural network models and many downstream applications. Nonetheless, as unique data properties have inspired distinct and powerful learning principles, this paper aims to explore their potentials towards mitigating adversarial inputs. In particular, our reveal the importance of using the temporal dependency in audio data to gain discriminate power against adversarial examples. Tested on the automatic speech recognition (ASR) tasks and three recent audio adversarial attacks, we find that (i) input transformation developed from image adversarial defense provides limited robustness improvement and is subtle to advanced attacks; (ii) temporal dependency can be exploited to gain discriminative power against audio adversarial examples and is resistant to adaptive attacks considered in our experiments. Our not only show promising means of improving the robustness of ASR systems, but also offer novel insights in exploiting domain-specific data properties to mitigate negative effects of adversarial examples. Deep Neural Networks (DNNs) have been widely adopted in a variety of machine learning applications BID18 BID20. However, recent work has demonstrated that DNNs are vulnerable to adversarial perturbations BID32 BID10. An adversary can add negligible perturbations to inputs and generate adversarial examples to mislead DNNs, first found in image-based machine learning tasks BID10 BID2 BID21 BID7 a; BID30.Beyond images, given the wide application of DNN-based audio recognition systems, such as Google Home and Amazon Alexa, audio adversarial examples have also been studied recently BID0 BID8 BID17. Comparing between image and audio learning tasks, although their state-of-the-art DNN architectures are quite different (i.e., convolutional v.s. recurrent neural networks), the attacking methodology towards generating adversarial examples is fundamentally unanimous -finding adversarial perturbations through the lens of maximizing the training loss or optimizing some designed attack objectives. For example, the same attack loss function proposed in BID8 ) is used to generate adversarial examples in both visual and speech recognition models. Nonetheless, different types of data usually possess unique or domain-specific properties that can potentially be used to gain discriminative power against adversarial inputs. In particular, the temporal dependency in audio data is an innate characteristic that has already been widely adopted in the machine learning models. However, in addition to improving learning performance on natural audio examples, it is still an open question on whether or not the temporal dependency can be exploited to help mitigate negative effects of adversarial examples. The focus of this paper has two folds. First, we investigate the robustness of automatic speech recognition (ASR) models under input transformation, a commonly used technique in the image domain to mitigate adversarial inputs. Our experimental show that four implemented transformation techniques on audio inputs, including waveform quantization, temporal smoothing, down-sampling and autoencoder reformation, provide limited robustness improvement against the recent attack method proposed in BID1, which aims to circumvent the gradient obfuscation issue incurred by input transformations. Second, we demonstrate that temporal dependency can be used to gain discriminative power against adversarial examples in ASR. We perform the proposed temporal dependency method on both the LIBRIS BID11 and Mozilla Common Voice datasets against three state-of-the-art attack methods BID0 BID36 considered in our experiments and show that such an approach achieves promising identification of non-adaptive and adaptive attacks. Moreover, we also verify that the proposed method can resist strong proposed adaptive attacks in which the defense implementations are known to an attacker. Finally, we note that although this paper focuses on the case of audio adversarial examples, the methodology of leveraging unique data properties to improve model robustness could be naturally extended to different domains. The promising also shed new lights in designing adversarial defenses against attacks on various types of data. Related work An adversarial example for a neural network is an input x adv that is similar to a natural input x but will yield different output after passing through the neural network. Currently, there are two different types of attacks for generating audio adversarial examples: the Speech-toLabel attack and the Speech-to-Text attack. The Speech-to-Label attack aims to find an adversarial example x adv close to the original audio x but yields a different (wrong) label. To do so, Alzantot et al. proposed a genetic algorithm BID0, and Cisse et al. proposed a probabilistic loss function BID8. The Speech-to-Text attack requires the transcribed output of the adversarial audio to be the same as the desired output, which has been made possible by BID16. Yuan et al. demonstrated the practical "wav-to-API" audio adversarial attacks BID36. Another line of research focuses on adversarial training or data augmentation to improve model robustness BID28 BID26 BID29 BID31, which is beyond our scope. Our proposed approach focuses on gaining the discriminative power against adversarial examples through embedded temporal dependency, which is compatible with any ASR model and does not require adversarial training or data augmentation. TO AUDIO DOMAIN? Although in recent years both image and audio learning tasks have witnessed significant breakthroughs accomplished by advanced neural networks, these two types of data have unique properties that lead to distinct learning principles. In images, the pixels entail spatial correlations corresponding to hierarchical object associations and color descriptions, which are leveraged by the convolutional neural networks (CNN) for feature extraction. In audios, the waveforms possess apparent temporal dependency, which is widely adopted by the recurrent neural networks (RNNs). For the segmentation task in the image domain, spatial consistency has played an important role in improving model robustness BID22. However, it remains unknown whether temporal dependency can have a similar effect of improving model robustness against audio adversarial examples. In this paper, we aim to address the following fundamental questions: (a) do lessons learned from image adversarial examples transfer to the audio domain?; and (b) can temporal dependency be used to discriminate audio adversarial examples? Moreover, studying the discriminative power of temporal dependency in audios not only highlights the importance of using unique data properties towards building robust machine learning models but also aids in devising principles for investigating more complex data such as videos (spatial + temporal properties) or multimodal cases (e.g., images + texts).Here we summarize two primary findings concluded from our experimental in Section 4.Audio input transformation is not effective against adversarial attacks Input transformation is a widely adopted defense technique in the image domain, owing to its low operating cost and easy integration with the existing network architecture BID24 BID33 BID9. Generally speaking, input transformation aims to perform certain feature transformation on the raw image in order to disrupt the adversarial perturbations before passing it to a neural network. Popular approaches include bit quantization, image filtering, image reprocessing, and autoencoder reformation BID35 BID12 BID25. However, many existing methods are shown to be bypassed by subsequent or adaptive adversarial attacks BID3 BID13 BID4 BID23. Moreover, Athalye et al. BID1 has pointed out that input transformation may cause obfuscated gradients when generating adversarial examples and thus gives a false sense of robustness. They also demonstrated that in many cases this gradient obfuscation issue can be circumvented, making input transformation still vulnerable to adversarial examples. Similarly, in our experiments we find that audio input transformations based on waveform quantization, temporal filtering, signal downsampling or autoencoder reformation suffers from similar weakness: the tested model with input transformation becomes fragile to adversarial examples when one adopts the attack considering gradient obfuscation as in BID1.Temporal dependency possesses strong discriminative power against adversarial examples in automatic speech recognition Instead of input transformation, in this paper, we propose to exploit the inherent temporal dependency in audio data to discriminate adversarial examples. Tested on the automatic speech recognition (ASR) tasks, we find that the proposed methodology can effectively detect audio adversarial examples while minimally affecting the recognition performance on normal examples. In addition, experimental show that a considered adaptive adversarial attack, even when knowing every detail of the deployed temporal dependency method, cannot generate adversarial examples that bypass the proposed temporal dependency-based approach. Combining these two primary findings, we conclude that the weakness of defense techniques identified in the image case is very likely to be transferred to the audio domain. On the other hand, exploiting unique data properties to develop defense methods, such as using temporal dependency in ASR, can lead to promising defense approaches that can resist adaptive adversarial attacks. In this section, we will introduce the effect of basic input transformations on audio adversarial examples, and analyze temporal dependency in audio data. We will also show that such temporal dependency can be potentially leveraged to discriminate audio adversarial examples. Inspired by image input transformation methods and as a first attempt, we applied some primitive signal processing transformations to audio inputs. These transformations are useful, easy to implement, fast to operate and have delivered several interesting findings. Quantization: By rounding the amplitude of audio sampled data into the nearest integer multiple of q, the adversarial perturbation could be disrupted since its amplitude is usually small in the input space. We choose q = 128, 256, 512, 1024 as our parameters. Local smoothing: We use a sliding window of a fixed length for local smoothing to reduce the adversarial perturbation. For an audio sample x i, we consider the K − 1 samples before and after it, denoted by [x i−K+1, . . ., x i, . . ., x i+K−1], as a local reference sequence and replace x i by the smoothed value (average, median, etc) of its reference sequence. Downsampling: Based on sampling theory, it is possible to down-sample a band-limited audio file without sacrificing the quality of the recovered signal while mitigating the adversarial perturbations in the reconstruction phase. In our experiments, we down-sample the original 16kHz audio data to 8kHz and then perform signal recovery. Autoencoder: In adversarial image defending field, the MagNet defensive method BID25 is an effective way to remove adversarial noises: Implement an autoencoder to project the adversarial input distribution space into the benign distribution. In our experiments, we implement a sequence-to-sequence autoencoder and the whole audio will be cut into frame-level pieces passing through the autoencoder and concatenate them in the final stage, while using the whole audio passing the autoencoder directly is proved to be ineffective and hard to utilize the underlying information. Due to the fact that audio sequence has an explicit temporal dependency (e.g., correlations in consecutive waveform segments), here we aim to explore if such temporal dependency will be affected by adversarial perturbations. The pipeline of the temporal dependency based method is shown in FIG0. Given an audio sequence, we propose to select the first k portion of it (i.e., the prefix of length k) as input for ASR to obtain transcribed as S k. We will also insert the whole sequence into ASR and select the prefix of length k of the transcribed as S {whole,k}, which has the same length as S k. We will then compare the consistency between S k and S {whole,k} in terms of temporal dependency distance. Here we adopt the word error rate (WER) as the distance metric BID19. For normal/benign audio instance, S k and S {whole,k} should be similar since the ASR model is consistent for different sections of a given sequence due to its temporal dependency. However, for audio adversarial examples, since the added perturbation aims to alter the ASR output toward the targeted transcription, it may fail to preserve the temporal information of the original sequence. Therefore, due to the loss of temporal dependency, S k and S {whole,k}, in this case, will not be able to produce consistent . Based on such hypothesis, we leverage the prefix of length k of the transcribed and the transcribed k portion to potentially recognize adversarial inputs. The presentation flows of the experimental are summarized as follows. We will first introduce the datasets, target learning models, attack methods, and evaluation metrics for different defense/detection methods that we focus on. We then discuss the defense/detection effectiveness for different methods against each attack respectively. Finally, we evaluate strong adaptive attacks against these defense/detection methods. We show that due to different data properties, the autoencoder based defense cannot effectively recover the ground truth for adversarial audios and may also have negative effects on benign instances as well. Input transformation is less effective in defending adversarial audio than images. In addition, even when some input transformation is effective for recovering some adversarial audio data, we find that it is easy to perform adaptive attacks against them. The proposed TD method can effectively detect adversarial audios generated by different attacks targeting on various learning tasks (classification and speech-to-text translation). In particular, we propose different types of strong adaptive attacks against the TD detection method. We show that these strong adaptive attacks are not able to generate effective adversarial audio against TD and we provide some case studies to further understand the performance of TD. In our experiments, we measure the effectiveness on several adversarial audio generation methods. For audio classification attack, we used Speech Commands dataset. For the speech-to-text attack, we benchmark each method on both LibriSpeech and Mozilla Common Voice dataset. In particular, for the Commander Song attack BID36, we measure the generated adversarial audios given by the authors. Dataset LibriSpeech dataset: LibriSpeech BID27 is a corpus of approximately 1000 hours of 16Khz English speech derived from audiobooks from the LibriVox project. We used samples from its test-clean dataset in their website and the average duration is 4.294s. We generated adversarial examples using the attack method in.Mozilla Common Voice dataset: Common Voice is a large audio dataset provided by Mozilla. This dataset is public and contains samples from human speaking audio files. We used the 16Khz-sampled data released in, whose average duration is 3.998s. The first 100 samples from its test dataset are used to mount attacks, which is the same attack experimental setup as in.Speech Commands dataset: Speech Commands dataset BID34 is an audio dataset contains 65000 audio files. Each audio is just a single command lasting for one second. Commands are "yes", "no", "up", "down", "left", "right", "on", "off", "stop", and "go".Model and learning tasks For the speech-to-text task, we use DeepSpeech speech-to-text transcription network, which is a biRNN based model with beam search to decode text. For audio classification task, we use a convolutional speech commands classification model. For the Command Song attack, we evaluate the performance on Kaldi speech recognition platform. Genetic algorithm based attack against audio classification (GA): For the audio classification task, we consider the state-of-the-art attack proposed in BID0. Here an audio classification model is attacked and the audio classes include "yes, no, up, down, etc.". They aimed to attack such a network to misclassify an adversarial instance based on either targeted or untargeted attack. Commander Song attack against speech-to-text translation (Commander): Commander Song BID36 ) is a speech-to-text targeted attack which can attack audio extracted from popular songs. The adversarial audio can even be played over the air with its adversarial characteristics. Since the Commander Song codes are not available, we measure the effectiveness of the generated adversarial audios given by the authors. Optimization based attack against speech-to-text translation (Opt): We consider the targeted speechto-text attack proposed by, which uses CTC-loss in a speech recognition system as an objective function and solves the task of adversarial attack as an optimization problem. Evaluation Metrics For defense method such as input transformation, since it aims to recover the ground truth (original instances) from adversarial instances, we use the word error rate (WER) and character error rate (CER) BID19 as evaluation metrics to measure the recovery efficiency. WER and CER are commonly used metrics to measure the error between recovered text and the ground truth in word level or character level. Generally speaking, the error rate (ER) is defined by ER = S+D+I N, where S, D, I is the number of substitutions, deletions and insertions calculated by dynamic string alignment, and N is the total number of word/character in the ground truth text. To fairly evaluate the effectiveness of these transformations against speech-to-text attack, we also report the ratio of translation distance between instance and corresponding ground truth before and after transformation. For instance, as a controlled experiment, given an audio instance x (adversarial instance is denoted as x adv), its corresponding ground truth y, and the ASR function g(·), we calculate the effectiveness ratio for benign instances as DISPLAYFORM0,y), where T (·) denotes the of transformation and D(·, ·) characterizes the distance function (WER and CER in our case). For adversarial audio, we calculate the similar efficiency ratio as R adv = D(g(T (x adv)),y) D(g(x adv),y). For the detection method, the standard evaluation metric is the area under curve (AUC) score, aiming to evaluate the detection efficiency. The proposed TD method is the first data-specific metric to detect adversarial audio, which focuses on how many adversarial instances are captured (true positive) without affecting benign instances (false positive). Therefore, we follow the standard criteria and report AUC for TD. For the proposed TD method, we compare the temporal dependency based on WER, CER, as well as the longest common prefix (LCP). LCP is a commonly used metric to evaluate the similarity between two strings. Given strings b 1 and b 2, the corresponding LCP is defined as max In this section, we measured our defense method of autoencoder based defense and input transformation defense for classification attack (GA) and speech-to-text attack (Commander and Opt). We summarize our work in TAB1 and list some basic . For Commander, due to unreleased training data, we are not able to train an autoencoder. For GA and Opt we have sufficient data to train autoencoder. Here we perform the primitive input transformation for audio classification targeted attacks and evaluate the corresponding effects. Due to the space limitation, we defer the of untargeted attacks to the supplemental materials. GA We first evaluate our input transformation against the audio classification attack (GA) in BID0. We implemented their attack with 500 iterations and limit the magnitude of adversarial perturbation within 5 (smaller than the quantization we used in transformation) and generated 50 adversarial examples per attack task (more targets are shown in the supplementary material). The attack success rate is 84% on average. For the ease of illustration, we use Quantization-256 as our input transformation. As observed in FIG2, the attack success rates decreased to only 2.1%, and 63.8% of the adversarial instances have been converted back to their original (true) label. We also measure the possible effects on original audio due to our transformation methods: the original audio classification accuracy without our transformation is 89.2%, and the rate slightly decreased to 89.0% after our transformation, which means the effects of input transformation on benign instances are negligible. In addition, it also shows that for classification tasks, such input transformation is more effective in mitigating the negative effects of adversarial perturbation. This potential reason could be that classification tasks do not rely on audio temporal dependency but focus on local features, while speech-to-text task will be harder to defend based on the tested input transformations. Commander We also evaluate our input transformation method against the Commander Song attack BID36, which implemented an Air-to-API adversarial attack. In the paper, the authors reported 91% attack detection rate using their defense method. We measured our Quan-256 input transformation on 25 adversarial examples obtained via personal communications. Based on the same detection evaluation metric in BID36 1, Quan-256 attains 100% detection rate for characterizing all the adversarial examples. Opt Here we consider the state-of-the-art audio attack proposed in. We separately choose 50 audio files from two audio datasets (Common Voice, LIBRIS) and generate attacks based on the CTC-loss. We evaluate several primitive signal processing methods as input transformation under WER and CER metrics in TAB1 and A2. We then also evaluate the WER and CER based effectiveness ratio we mentioned before to Quantify the effectiveness of transformation. R benign are shown in the brackets for the first two columns in TAB1 and A2, while R adv is shown in the brackets of last two columns within those tables. We compute our using both ground truth and adversarial target "This is an adversarial example" as references. Here small R benign which is close to 1 indicates that transformation has little effect on benign instances, small R adv represents transformation is effective recovering adversarial audio back to benign. From Tables A1 and A2 we showed that most of the input transformations (e.g., Median-4, Downsampling and Quan-256) effectively reduce the adversarial perturbation without affecting the original audio too much. Although these input transformations show certain effectiveness in defending against adversarial audios, we find that it is still possible to generate adversarial audios by adaptive attacks in Section 4.4. Towards defending against (non-adaptive) adversarial images, MagNet BID25 has achieved promising performance by using an autoencoder to mitigate adversarial perturbation. Inspired by it, here we apply a similar autoencoder structure for audio and test if such input transformation can be applied to defending against adversarial audio. We apply a MagNet-like method for feature-extracted audio spectrum map: we build an encoder to compress the information of origin audio features into latent vector z, then use z for reconstruction by passing through another decoder network under frame level and combine them to obtain the transformed audio BID15. Here we analyzed the performance of Autoencoder transformation in both GA and Opt attack. We find that MagNet which gained great effectiveness on defending adversarial images in the oblivious attack setting BID4 BID23, has limited effect on the audio defense. GA We presented our in TAB1 that against classification attack, Autoencoder did not perform well by only reducing attack success rate to 8.2% defeat by other input transformation methods. Since you can reduce the attack success rate to 10% by just destroying the origin audio data and altering to random guess, it's hard to say that Autoencoder method has good performance. Opt We report that the autoencoder works not very well for transforming benign instances (57.6 WER in Common Voice compared to 27.5 WER without transformation, 30.0 WER in LIBRIS compared to 12.4 WER without transformation), also fails to recover adversarial audio (76.5 WER in Common Voice and 99.4 WER in LIBRIS). This shows that the non-adaptive additive adversarial perturbation can bypass the MagNet-like autoencoder on audio, which implies different robustness implications of image and audio data. In this section, we will evaluate the proposed TD detection method on different attacks. We will first report the AUC for detecting different attacks with TD to demonstrate the effectiveness, and we will provide some additional analysis and examples to help better understand TD. We only evaluate our TD method on speech-to-text attacks (Commander and Opt) because of the audio in the Speech Commands dataset for classification attack is just a single command lasting for one second and thus its temporal dependency is not obvious. Commander In Commander Song attack, we directly examine whether the generated adversarial audio is consistent with its prefix of length k or not. We report that by using TD method with k = 1 2, all the generated adversarial samples showed inconsistency and thus were successfully detected. Opt Here we show the empirical performance of distinguishing adversarial audios by leveraging the temporal dependency of audio data. In the experiments, we use these three metrics, WER, CER and LCP, to measure the inconsistency between S k and S {whole,k}. As a baseline, we also directly train a one layer LSTM with 64 hidden feature dimensions based on the collected adversarial and benign audio instances for classification. Some examples of translated for benign and adversarial audios are shown in TAB2. Here we consider three types of adversarial targets: short -hey google; medium -this is an adversarial example; and long -hey google please cancel my medical appointment. We report the AUC score for these detection for k = 1/2 in TAB3. We can see that by using WER as the detection metric, the temporal dependency based method can achieve AUC as high as 0.936 on Common Voice and 0.93 on LIBRIS. We also explore different values of k and we observe that the do not vary too much (detailed can be found in Table A6 in Appendix). When k = 4/5, the AUC score based on CER can reach 0.969, which shows that such temporal dependency based method is indeed promising in terms of distinguishing adversarial instances. Interestingly, these suggest that the temporal dependency based method would suggest an easy-implemented but effective method for characterizing adversarial audio attacks. In this section, we measured some adaptive attack against the defense and detection methods. Since the autoencoder based defense almost fails to defend against different attacks, here we will focus on the input transformation based defense and TD detection. Given that Opt is the strongest attack here, we will mainly apply Opt to perform adaptive attack against the speech-to-text translation task. We list our experiments' structure in TAB4. For full please refer to the Appendix. Here we apply adaptive attacks against the preceding input transformations and therefore evaluate the robustness of the input transformation as defenses. We implemented our adaptive attack based on three input transformation methods: Quantization, Local smoothing, and Downsampling. For these transformations, we leverage a gradientmasking aware approach to generate adaptive attacks. In the optimization based attack, the attack achieved by solving the optimization problem: min δ δ 2 2 + c · l(x + δ, t), where δ is referred to the perturbation, x the benign audio, t the target phrase, and l(·) the CTC-loss. Parameter c is iterated to trade off the importance of being adversarial and remaining close to the original instance. For quantization transformation, we assume the adversary knows the quantization parameter q. We then change our attack targeted optimization function to min δ qδ 2 2 + c · l(x + qδ, t). After that, all the adversarial audios can be resistant against quantization transformations and it only increased a small magnitude of adversarial perturbation, which can be ignored by human ears. When q is large enough, the distortion would increase but the transformation process is also ineffective due to too much information loss. For downsampling transformation, the adaptive attack is conducted by performing the attack on the sampled elements of origin audio sequence. Since the whole process is differentiable, we can do adaptive attack through gradient directly and all the adversarial audios are able to attack. For local smoothing transformation, it is also differentiable in case of average smoothing transformation, so we can pass the gradient effectively. To attack against median smoothing transformation, we can just convert the gradient back to the median and update its value, which is similar to the maxpooling layer's backpropagation process. By implementing the adaptive attack, all the smoothing transformation is shown to be ineffective. We chose our samples randomly from LIBRIS and Common Voice audio dataset with 50 audio samples each. We implemented our adaptive attack on the samples and passed them through the corresponding input transformation. We use down-sampling from 16kHZ to 8kHZ, median / average smoothing with one-sided sequence length K = 4, quantization method with q = 256 as our input transformation methods. In, Decibels (a logarithmic scale that measures the relative loudness of an audio sample) is applied as the measurement of the magnitude of perturbation: dB(x) = max i 20 · log 10 (x i), which x referred as adversarial audio sampled sequence. The relative perturbation is calculated as dB x (δ) = dB(δ) − dB(x), where δ is the crafted adversarial noise. We measured our adaptive attack based on the same criterion. We show that all the adaptive attacks become effective with reasonable perturbation, as shown in Table 6. As suggested in, almost all the adversarial audios have distortion dB x (δ) from -15dB to -45dB which is tolerable to human ears. From Table 6, the added perturbation are mostly within this range. Adaptive Attacks Against Temporal Dependency Based Method To thoroughly evaluate the robustness of temporal dependency based method, we also perform some strong adaptive attack against it. Notably, even if the adversary knows k, the adaptive attack is hard to conduct due to the fact that this process is non-differentiable. Therefore, we propose three types of strong adaptive attacks here aiming to explore the robustness of the temporal consistency based method. Segment attack: Given the knowledge of k, we first split the audio into two parts: the prefix of length k of the audio S k and the rest S k −. We then apply a similar attack to add perturbation to only S k. We hope this audio can be attacked successfully without changing S k − since the second part would not receive gradient updates. Therefore, when performing the temporal-based consistency check, T (S k) would be translated consistently with T (S {whole,k} ). To maximally leverage the information of k, here we propose two ways to attack both S k and S k − individually, and then concatenate them together.1. the target of S k is the first k−portion of the adversarial target, and S k − is attacked to the rest.2. the target of S k is the whole adversarial target, while we attack S k − to be silence, which means S k − transcribing nothing. This is different from the segment attack where S k − is not modified at all. Combination attack: To balance the attack success rate for both sections and the whole sentence against TD, we apply the attack objective function as min δ δ 2 2 + c · (l(x + δ, t) + l((x + δ) k, t k ), where x refers to the whole sentence. For segment attack, we found that in most cases the attack cannot succeed, that the attack success rate remains at 2% for 50 samples in both LIBRIS and Common Voice datasets, and some of the -35.65 -20.91 -9.48 -23.42 -25.12 examples are shown in Appendix. We conjecture the reasons as: 1. S k alone is not enough to be attacked to the adversarial target due to the temporal dependency; 2. the speech recognition on S k − cannot be applied to the whole recognition process and therefore break the recognition process for S k.For concatenation attack, we also found that the attack itself fails. That is, the transcribed of adv(S k)+adv(S k −) differs from the translation of S k +S k −. Some examples are shown in Appendix. The failure of the concatenation adaptive attack more explicitly shows that the temporal dependency plays an important role in audio. Even if the separate parts are successfully attacked into the target, the concatenated instance will again totally break the perturbation and therefore render the adaptive attack inefficient. On the contrary, such concatenation will have negligible effects on benign audio instances, which provides a promising direction to detect adversarial audio. For combination attack, we vary the section portion k D used by TD and evaluate the cases where the adaptive attacker uses the same/different section k A. We define Rand(a,b) as uniformly sampling from [a,b]. We consider a stronger attacker, for whom the k A can be a set containing random sections. The detection for different settings are shown in TAB5. From the , we can see that when |k A | = 1, if the attacker uses the same k A as k D to perform the adaptive attack, the attack can achieve relatively good performance and if the attacker uses different k A, the attack will fail with AUC above 85%. We also evaluate the case that defender randomly sample k D during the detection and find that it's very hard for the adaptive attacker to perform attacks, which can improve model robustness in practice. For |k A | > 1, the attacker can achieve some attack success when the set contains k D. But when |k A | increases, the attacker's performance becomes worse. The complete are given in the Appendix. Notably, the random sample based TD appears to be robust in all cases. This paper proposes to exploit the temporal dependency property in audio data to characterize audio adversarial examples. Our experimental show that while four primitive input transformations on audio fail to withstand adaptive adversarial attacks, temporal dependency is shown to be resistant to these attacks. We also demonstrate the power of temporal dependency for characterizing adversarial examples generated by three state-of-the-art audio adversarial attacks. The proposed method is easy to operate and does not require model retraining. We believe our shed new lights in exploiting unique data properties toward adversarial robustness. This work is partially supported by DARPA grant 00009970. Figure

Adversarial audio discrimination using temporal dependency

Inspired by trial-to-trial variability in the brain that can result from multiple noise sources, we introduce variability through noise in the knowledge distillation framework and studied their effect on generalization and robustness.


Question: Represent the scientific article to find a short TLDR!

Question: In order to alleviate the notorious mode collapse phenomenon in generative adversarial networks (GANs), we propose a novel training method of GANs in which certain fake samples can be reconsidered as real ones during the training process. This strategy can reduce the gradient value that generator receives in the region where gradient exploding happens. We show that the theoretical equilibrium between the generators and discriminations actually can be seldom realized in practice. And this in an unbalanced generated distribution that deviates from the target one, when fake datepoints overfit to real ones, which explains the non-stability of GANs. We also prove that, by penalizing the difference between discriminator outputs and considering certain fake datapoints as real for adjacent real and fake sample pairs, gradient exploding can be alleviated. Accordingly, a modified GAN training method is proposed with a more stable training process and a better generalization. Experiments on different datasets verify our theoretical analysis. In the past few years, Generative Adversarial Networks (GANs) have been one of the most popular topics in generative models and achieved great success in generating diverse and high-quality images recently (; ;). GANs are powerful tools for learning generative models, which can be expressed as a zero-sum game between two neural networks. The generator network produces samples from the arbitrary given distribution, while the adversarial discriminator tries to distinguish between real data and generated data. Meanwhile, the generator network tries to fool the discriminator network by producing plausible samples which are close to real samples. When a final theoretical equilibrium is achieved, discriminator can never distinguish between real and fake data. However, we show that a theoretical equilibrium often can not be achieved with discrete finite samples in datasets during the training process in practice. Although GANs have achieved remarkable progress, numerous researchers have tried to improve the performance of GANs from various aspects;;; ) because of the inherent problem in GAN training, such as unstability and mode collapse. showed that a theoretical generalization guarantee does not be provided with the original GAN objective and analyzed the generalization capacity of neural network distance. The author argued that for a low capacity discriminator, it can not provide generator enough information to fit the target distribution owing to lack of ability to detect mode collapse. argued that poor generation capacity in GANs comes from discriminators trained on discrete finite datasets ing in overfitting to real data samples and gradient exploding when generated datapoints approach real ones. As a , proposed a zero-centered gradient penalty on linear interpolations between real and fake samples (GAN-0GP-interpolation) to improve generalization capability and prevent mode collapse ed from gradient exploding. Recent work further studied generalization from a new perspective of privacy protection. In this paper, we focus on mode collapse ed from gradient exploding studied in and achieve a better generalization with a much more stable training process. Our contributions are as follows: discriminator with sigmoid function in the last layer removed D r = {x 1, · · ·, x n} the set of n real samples D g = {y 1, · · ·, y m} the set of m generated samples D f = {f 1, · · ·, f m} the candidate set of M 1 generated samples to be selected as real D F AR ⊂ {f 1, · · ·, f m} the set of M 0 generated samples considered as real 1. We show that a theoretical equilibrium, when optimal discriminator outputs a constant for both real and generated data, is unachievable for an empirical discriminator during the training process. Due to this fact, it is possible that gradient exploding happens when fake datapoints approach real ones, ing in an unbalanced generated distribution that deviates from the target one. 2. We show that when generated datapoints are very close to real ones in distance, penalizing the difference between discriminator outputs and considering fake as real can alleviate gradient exploding to prevent overfitting to certain real datapoints. 3. We show that when more fake datapoints are moved towards a single real datapoint, gradients of the generator on fake datapoints very close to the real one can not be reduced, which partly explains the reason of a more serious overfitting phenomenon and an increasingly unbalanced generated distribution. 4. Based on the zero-centered gradient penalty on data samples (GAN-0GP-sample) proposed in , we propose a novel GAN training method by considering some fake samples as real ones according to the discriminator outputs in a training batch to effectively prevent mode collapse. Experiments on synthetic and real world datasets verify that our method can stabilize the training process and achieve a more faithful generated distribution. In the sequel, we use the terminologies of generated samples (datapoints) and fake samples (datapoints) indiscriminately. Tab. 1 lists some key notations used in the rest of the paper. Unstability. GANs have been considered difficult to train and often play an unstable role in training process. Various methods have been proposed to improve the stability of training. A lot of works stabilized training with well-designed structures (; ; ;) and utilizing better objectives (; ; ;). Gradient penalty to enforce Lipschitz continuity is also a popular direction to improve the stability including Gulrajani et al.,Petzka et al. (2018, Roth et al.,Qi (2017 . From the theoretical aspect, showed that GAN optimization based on gradient descent is locally stable and proved local convergence for simplified zero-centered gradient penalties under suitable assumptions. For a better convergence, a two time-scale update rule (TTUR) and exponential moving averaging (EMA) (Yazıcı et al. ) have also been studied. Mode collapse. Mode collapse is another persistent essential problem for the training of GANs, which means lack of diversity in the generated samples. The generator may sometimes fool the discriminator by producing a very small set of high-probability samples from the data distribution. Recent work studied the generalization capacity of GANs and showed that the model distributions learned by GANs do miss a significant number of modes. A large number of ideas have been proposed to prevent mode collapse. Multiple generators are applied in , , to achieve a more faithful distribution. Mixed samples are considered as the inputs of discriminator in , to convey information on diversity. Recent work studied mode collapse from probabilistic treatment and from entropy of distribution. In the original , the discriminator D maximizes the following objective: The logistic sigmoid function σ(x) = 1 1+e −x is usually used in practice, leading to and to prevent gradient collapse, the generator G maximizes where D 0 is usually represented by a neural network. showed that the optimal discriminator D in Eqn.1 is As the training progresses, p g will be pushed closer to p r. If G and D are given enough capacity, a global equilibrium is reached when p r = p g, in which case the best strategy for D on supp(p r) ∪ supp(p g) is just to output 1 2 and the optimal value for Eqn.1 is 2 log(1 2). With finite training examples in training dataset D r in practice, an empirical version is applied to approximate Eqn.1, using, where x i is from the set D r of n real samples and y i is from the set D g of m generated samples, respectively. Mode collapse in the generator is attributed to gradient exploding in discriminator, according to. When a fake datapoint y 0 is pushed to a real datapoint x 0 and if |D(x 0)− D(y 0)| ≥, is satisfied, the absolute value of directional derivative of D in the direction µ = x 0 − y 0 will approach infinity leading to gradient exploding: Since the gradient ∇ y0 D(y 0) at y 0 outweights gradients towards other modes in a mini-batch, gradient exploding at datapoint y 0 will move multiple fake datapoints towards x 0 ing in mode collapse. Theoretically discriminator outputs a constant 1 2 when a global equilibrium is reached. However in practice, discriminator can often easily distinguish between real samples and fake samples , making a theoretical equilibrium unachievable. Because the distribution p r of real data is unknown for discriminator, discriminator will always consider datapoints in the set D r of real samples as real while D g of generated samples as fake. Even when generated distribution p g is equivalent to the target distribution p r, D r and D g is disjoint with probability 1 when they are sampled from two continuous distributions respectively (proposition 1 in). In this case, actually p g is pushed towards samples in D r instead of the target distribution. However, we show next because of the fact of an unachievable theoretical equilibrium for empirical discriminator during the training process, an unbalanced distribution would be generated that deviates from the target distribution. Proposition 1. For empirical discriminator in original GAN, unless p g is a discrete uniform distribution on D r, the optimal discriminator D output on D r and D g is not a constant 1 2, since there exists a more optimal discriminator which can be constructed as a MLP with O(2d x) parameters. See Appendix A for the detailed proof. If all the samples in D r can be remembered by discriminator and generator, and only if generated samples can cover D r uniformly, which means D g = D r, a theoretical equilibrium in discriminator can be achieved. However, before generator covers all the samples in D r uniformly during the training process, the fact of an unachievable theoretical equilibrium makes it possible that there exists a real datapoint x 0 with a higher discriminator output than that of a generated datapoint y 0. When y 0 approaches x 0 very closely, gradient exploding and overfitting to a single datapoint happen, ing an unbalanced distribution and visible mode collapse. See the generated on a Gaussian dataset of original GAN in Fig. 1a and 1e. The generated distribution neither covers the target Gaussian distribution nor fits all the real samples in D r, making an unbalanced distribution visible. Furthermore, in practice discriminator and generator are represented by a neural network with finite capacity and dataset D r is relatively huge, generator can never memorize every discrete sample ing in a theoretical equilibrium unachievable. In the following subsections, we are interested in the way of stabilizing the output of discriminator to alleviate gradient exploding to achieve a more faithful generated distribution. Let's first consider a simplified scenario where a fake datapoint y 0 is close to a real datapoint x 0. Generator updates y 0 according to the gradient that the generator receives from the discriminator with respect to the fake datapoint y 0, which can be computed as: When y 0 approaches x 0 very closely and a theoretical discriminator equilibrium is not achieved here, namely D 0 (x 0) − D 0 (y 0) ≥, the absolute value of directional derivative (∇ µ D 0) y0 in the direction µ = x 0 − y 0 at y 0 tends to explode and will outweigh directional derivatives in other When y 0 is very close to x 0, the norm of the gradient generator receives from the discriminator with respect to y 0 can be computed as: If y 0 is in the neighborhood of x 0, i.e.,, where δ is a small positive value, we call {x 0, y 0} a close real and fake pair. We are interested in reducing the approximated value of the gradient for a fixed pair {x 0, y 0} to prevent multiple fake datapoints overfitting to a single real one. Note that the output of D 0 for real datapoint x 0 has a larger value than that of fake datapoint y 0. So for a fixed pair {x 0, y 0}, when D 0 (y 0) increases and D 0 (x 0) − D 0 (y 0) decreases, the target value decreases. And, when D 0 (y 0) decreases and D 0 (x 0) − D 0 (y 0) increases, the target value increases, according to Eqn. 6. Now we consider a more general scenario where for a real datapoint x 0, in a set of n real samples, there are m 0 generated datapoints {y 1, y 2, · · ·, y m0} very close to x 0 in the set of m generated samples. We are specially interested in the optimal discriminator output at x 0 and {y 1, y 2, · · ·, y m0}. For simplicity, we make the assumption that discriminator outputs at these interested points are not affected by other datapoints in D r and D g. We also assume discriminator has enough capacity to achieve optimum in this local region. However, without any constraint, discriminator will consistently enlarge the gap between outputs for real datapoints and that for generated ones. Thus, an extra constraint is needed to alleviate the difference between discriminator outputs on real and fake datapoints. It comes naturally to penalize the L 2 norm of D 0 (x 0) − D 0 (y i). Denoting the discriminator output for x 0, D 0 (x 0) as ξ 0 and D 0 (y i) as ξ i, i = 1, · · ·, m 0, we have the following empirical discriminator objective: where the interested term Based on Proposition 2, penalizing the difference between discriminator outputs on close real and fake pairs {x 0, y i} can reduce the norm of ∇ y i L G (y i) from Eqn.6, making it possible to move fake datapoints to other real datapoints instead of only being trapped at x 0. However in practice, it is hard to find the close real and fake pairs to penalize the corresponding difference between discriminator outputs. If we directly penalize the L 2 norm of D 0 (x i) − D 0 (y i) when {x i, y i} are not a pair of close datapoints, ||∇ y i L G (y i)|| for y i may even get larger. Consider D 0 (y i) has a higher value than D 0 (x i), which could happen when x i has more corresponding close fake datapoints than the real datapoint x yi corresponding to y i from Proposition 2. Direct penalization will make D 0 (y i) lower, then D 0 (x yi) − D 0 (y i) gets higher and ||∇ y i L G (y i)|| higher. Thus in practice we could enforce a zero-centered gradient penalty of the form ||(∇D 0) v || 2 to stabilize discriminator output, where v can be real datapoints or fake datapoints. Although thought that discriminator can have zero gradient on the training dataset and may still have gradient exploding outside the training dataset, we believe a zero-centered gradient penalty can make it harder for discriminator to distinguish between real and fake datapoints and fill the gap between discriminator outputs on close real and fake pairs to prevent overfitting to some extent. Fig. 1b and 1f alleviate overfitting phenomenon compared with no gradient penalty in Fig. 1a and 1e. proposed another zero-centered gradient penalty of the form ||(∇D 0) v || 2, where v is a linear interpolation between real and fake datapoints, to prevent gradient exploding. However, we consider it's not a very efficient method to fill the gap between discriminator outputs on close real and fake pairs. To begin with, the of direct linear interpolation between real and fake datapoints may not lie in supp(p r) ∪ supp(p g). Although the author also considered the interpolation on latent codes, it needs an extra encoder which increases operational complexity. Furthermore, for arbitrary pair of real and fake datapoints, the probability that linear interpolation between them lie where gradient exploding happens is close to 0, because large gradient happens when a fake datapoint approaches closely a real datapoint, ing in the gap between discriminator outputs on close real and fake pairs hard to fill. Based on Proposition 2, we also find that when more fake datapoints are moved to the corresponding real datapoint, ||∇ y i L G (y i)|| for a fake datpoint y i only to increase from Eqn.6. It means with the training process going on, more fake datapoints tend to be attracted to one single real datapoint and it gets easier to attract much more fake datapoints to the real one. It partly explains the unstability of GAN training process that especially during the later stage of training, similar generated samples are seen. Compared with Fig. 1a, 1b and 1c at iter.100,000, Fig. 1e, 1f and 1g at iter.200,000 have a worse generalization and much more similar samples are generated with the training process going on. In this subsection, we aim to make ||∇ y i L G (y i)||, i = 1, · · ·, m 0 smaller for optimal empirical discriminator by considering some fake as real on close real and fake pairs based on the above discussions. Suppose for each fake datapoint, it's considered as real datapoint with probability p 0 when training real datapoints, ing in the following empirical discriminator objective: where A is a binary random variable taking values in {0, 1} with Pr(A = 1) = p 0 and the interested term Note that only penalizing the difference between discriminator outputs on close real and fake pairs in Subsection 4.2 is just a special case of considering fake as real here when p 0 = 0. Based on Proposition 3, considering fake datapoints as real with increasing probability p 0 for real datapoints training part can reduce the norm of ∇ y i L G (y i) from Eqn.6. It means when we consider more fake as real where large gradient happens for real training part, the attraction to the real datapoint for fake ones can be alleviate to make it easier to be moved to other real datapoints and prevent the overfitting to one single real datapoint. Note that for a fixed p 0, when the number m 0 of fake datapoints very close to the real one increases, more fake datapoints will be considered as real to alleviate the influences of increasing m 0 discussed in Subsection 4.2. To overcome the problem of overfitting to some single datapoints and achieve a better generalization, we propose that fake samples generated by generator in real time can be trained as real samples in discriminator. For original N real samples in a training batch in training process, we substitute them with N 0 real samples in D r and M 0 generated samples in D g, where N = N 0 + M 0. Our approach is mainly aimed at preventing large gradient in the region where many generated samples overfit one single real sample. To find generated samples in these regions, we choose the generated ones with low discriminator output, owing to the reason that discriminator tends to have a lower output for the region with more generated datapoints approaching one real datapoint from Proposition 2. Therefore, we choose needed M 0 generated samples denoted as set D F AR as real samples from a larger generated set D f containing M 1 generated samples {f 1, f 2, · · ·, f M1} according to corresponding discriminator output: We also add a zero-centered gradient penalty on real datapoints based on the discussions in Subsection 4.2, ing in the following empirical discriminator objective in a batch containing N real samples and M fake samples: where In practice, we usually let N = M. Because some fake datapoints are trained as real ones, the zero-centered gradient penalty are actually enforced on the mixture of real and fake datapoints. When we sample more generated datapoints for D f to decide the needed M 0 datapoints as real, the probability of finding the overfitting region with large gradient is higher. When more fake datapoints in D F AR that are close to corresponding real ones are considered as real for training, it is equivalent to increase the value of p 0 in Subsection 4.3. For a larger D F AR, the number of real samples N 0 will decrease for a fixed batchsize N and the speed to cover real ones may be slowed at the beginning of training owning to the reason that some fake datapoints are considered as real and discriminator will be not so confident to give fake ones a large gradient to move them to real ones. Our method can stabilize discriminator output and prevent mode collapse caused by gradient exploding efficiently based on our theoretical analysis. A more faithful generated distribution will be achieved in practice. To test the effectiveness of our method in preventing an unbalanced distribution ed from overfitting to only some real datapoints, we designed a dataset with finite real samples coming from a Gaussian distributions and trained MLP based GANs with different gradient penalties and our method on that dataset. For gradient penalties in all GANs, the weight λ is set 10. Training batch is set 64 and one quarter of the real training batch are generated samples picked from 256 generated samples according to discriminator output, namely M 0 = 16 and M 1 = 256 in Eqn. 11. Learning rate is set 0.003 for both G and D. The is shown in Fig.1. It can be observed that original GAN, GAN-0GP-sample and GAN-0GP-interpolation all have serious overfitting problem leading to a biased generated distribution with training process going on, while our method can generate much better samples with good generalization. We also test our method on a mixture of 8 Gaussians dataset where random samples in different modes are far from each other. The evolution of our method is depicted in Fig.2. We observe that although our method only covers 3 modes at the beginning, it can cover other modes gradually because our method alleviates the gradient exploding on close real and fake datapoints. It is possible that fake datapoints are moved to other Gaussian modes when the attraction to other modes is larger than to the overfitted datapoints. Hence, our method has the ability to find the uncovered modes to achieve a faithful distribution even when samples in high dimensional space are far from each other. More synthetic experiments can be found in Appendix D. To test our method on real world data, we compare our method with GAN-0GP-sample on CIFAR-10 , CIFAR-100 and a more challenging dataset ImageNet with ResNet-architectures similar with that in. Inception score and FID are used as quantitative. The FID score is evaluated on 10k generated images and statistics of data are calculated at the same scale of generation. Better generation can be achieved with higher inception score and lower FID value. The maximum number of iterations for CIFAR experiment is 500k, while for ImageNet is 600k because of training difficulty with much more modes. We use the code from. The weight λ for gradient penalty is also set 10. Training batch is set 64 and for a better gradient alleviation on close real and fake datapoints, half of the real training batch are generated samples with M 0 = 32 and M 1 = 256 in Eqn. 11. For CIFAR experiments, we use the RMSProp optimizer with α = 0.99 and a learning rate of 10 −4. For ImageNet experiments, we use the Adam optimizer with α = 0, β = 0.9 and TTUR with learning rates of 10 −4 and 3 × 10 −4 for the generator and discriminator respectively. We use an exponential moving average with decay 0.999 over the weights to produce the final model. The on Inception score and FID are shown in Fig. 3 and 4. Our method outperforms GAN-0GP-sample by a large margin. As predicted in Section 5, the speed of our method to cover real ones could be slowed at the beginning of training with some fake considered as real. However, our method can cover more modes and have a much better balanced generation than the baseline. The losses of discriminator and generator during the process of CIFAR-10 training are shown in Fig.5. It can be observed that our method has a much more stable training process. Owing to the overfitting to some single datapoints and an unbalanced generated distribution missing modes, the losses of discriminator and generator for GAN-0GP-sample gradually deviate from the optimal theoretical value, namely 2 log 2 ≈ 1.386 for discriminator and log 2 ≈ 0.693 for generator respectively. However, our method has a much more stable output of discriminator to achieve the losses for discriminator and generator very close to theoretical case. It proves practically that our method can stabilize discriminator output on close real and fake datapoints to prevent more datapoints from (a) (b) Figure 6: Inception score and FID on ImageNet of GAN-0GP-sample and GAN-0GP-sample with our method trapped in a local region and has a better generalization. The losses of discriminator and generator on CIFAR-100 and image samples can be found in Appendix D. For the challenging ImageNet task, we train GANs to learn a generative model of all 1000 classes at resolution 64 × 64 with the limitation of our hardware. However, our models are completely unsupervised learning models with no labels used instead of another 256 dimensions being used in latent code z as labels in. The in Fig. 6 show that our methods still outperforms GAN-0GP-sample on ImageNet. Our method can produce samples of state of the art quality without using any category labels and stabilize the training process. Random selected samples and losses of discriminator and generator during the training process can be found in Appendix D. In this paper, we explain the reason that an unbalanced distribution is often generated in GANs training. We show that a theoretical equilibrium for empirical discriminator is unachievable during the training process. We analyze the affection on the gradient that generator receives from discriminator with respect to restriction on difference between discriminator outputs on close real and fake pairs and trick of considering fake as real. Based on the theoretical analysis, we propose a novel GAN training method by considering some fake samples as real ones according to the discriminator outputs in a training batch. Experiments on diverse datasets verify that our method can stabilize the training process and improve the performance by a large margin. For empirical discriminator, it maximizes the following objective: When p g is a discrete uniform distribution on D r, and generated samples in D g are the same with real samples in D r. It is obvious that the discriminator outputs 1 2 to achieve the optimal value when it cannot distinguish fake samples from real ones. For continues distribution p g, has proved that an -optimal discriminator can be constructed as a one hidden layer MLP with O(d x (m + n)) parameters, namely D(x) ≥ 1 2 + 2, ∀x ∈ D r and D(y) ≤ 1 2 − 2, ∀y ∈ D g, where D r and D g are disjoint with probability 1. In this case, discriminator objective has a larger value than the theoretical optimal version: So the optimal discriminator output on D r and D g is not a constant 1 2 in this case. Even discriminator has much less parameters than O(d x (m + n)), there exists a real datapoint x 0 and a generated datapoint y 0 satisfying D(x 0) ≥ 1 2 + 2 and D(y 0) ≤ 1 2 − 2. Whether p g is a discrete distribution only cover part samples in D r or a continues distribution, there exists a generated datapoint y 0 satisfying y 0 ∈ D r. Assume that samples are normalized: Let W 1 ∈ R 2×dx, W 2 ∈ R 2×2 and W 3 ∈ R 2 be the weight matrices, b ∈ R 2 offset vector and k 1,k 2 a constant, We can construct needed discriminator as a MLP with two hidden layer containing O(2d x) parameters. We set weight matrices For any input v ∈ D r ∪ D g, the discriminator output is computed as: where σ(x) = 1 1+e −x is the sigmoid function. Let α = W 1 v − b, we have where l < 1. Let β = σ(k 1 α), we have as k 2 → ∞. Hence, for any input v ∈ D r ∪ D g, discriminator outputs In this case, discriminator objective also has a more optimal value than the theoretical optimal version: So the optimal discriminator output on D r and D g is also not a constant 1 2 in this case. We rewrite f (ξ 0, ξ 1, · · ·, ξ m0) here To achieve the optimal value, let f (ξ i) = 0, i = 0, · · ·, m 0 and we have It is obvious that ξ 1 = ξ 2 = · · · = ξ m0 = ξ. Hence we have We can solve Substitute Eqn. 28 into Eqn. 26 and we get We can also have from Eqn. 28 and Eqn. 26 respectively Note that there must exist an optimal ξ 0 satisfying f (ξ 0) = 0 in Eqn. 29, so ξ 0 + ln(We rewrite h(ξ 0, ξ 1, · · ·, ξ m0) here Let f (ξ i) = 0, i = 0, · · ·, m 0 and we have ξ 1 = ξ 2 = · · · = ξ m0 = ξ, 1 − σ(ξ 0) − 2nk 0 (ξ 0 − ξ) = 0, We can solve ξ = − ln The derivative of g(ξ 0) with respect to ξ 0 is computed as g (ξ 0) > 0. Hence ξ * 0 increases with p 0 increasing. From Eqn. 33, we also have we further know that ξ * increases and ξ * 0 − ξ * decreases with p 0 increasing. (a) (b) Figure 7: Losses of discriminator (not including regularization term) and generator on CIFAR-100 of GAN-0GP-sample and GAN-0GP-sample with our method For synthetic experiment, the network architectures are the same with that in. While for real world data experiment, we use the similar architectures in. We use Pytorch for development.

Represent the TLDR for finding the corresponding scientific article.

We propose a novel GAN training method by considering certain fake samples as real to alleviate mode collapse and stabilize training process.

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.


Question: Represent the scientific article to find a short TLDR.

Question: We present a tool for Interactive Visual Exploration of Latent Space (IVELS) for model selection. Evaluating generative models of discrete sequences from a continuous latent space is a challenging problem, since their optimization involves multiple competing objective terms. We introduce a model-selection pipeline to compare and filter models throughout consecutive stages of more complex and expensive metrics. We present the pipeline in an interactive visual tool to enable the exploration of the metrics, analysis of the learned latent space, and selection of the best model for a given task. We focus specifically on the variational auto-encoder family in a case study of modeling peptide sequences, which are short sequences of amino acids. This task is especially interesting due to the presence of multiple attributes we want to model. We demonstrate how an interactive visual comparison can assist in evaluating how well an unsupervised auto-encoder meaningfully captures the attributes of interest in its latent space. Unsupervised representation learning and generation of text from a continuous space is an important topic in natural language processing. This problem has been successfully addressed by variational auto-encoders (VAE) BID16 and variations, which we will introduce in Section 2. The same methods are also relevant to areas like drug discovery, as the therapeutic small molecules and macromolecules (nucleic acids, peptides, proteins) can be represented as discrete linear sequences, analogous to text strings. Our case study of interest is modeling peptide sequences. In the VAE formulation, we define the sequence representation as a latent variable modeling problem of inputs x and latent variables z, where the joint distribution p(x, z) is factored as p(z)p θ (x|z) and the inference of the hidden variable z for a given input x is approximated through an inference network q φ (z|x). The auto-encoder training typically aims to minimize two competing objectives: (a) reconstruction of the input and (b) regularization in the latent space. Term (b) acts as a proxy to two real desiderata: (i) "meaningful" representations in latent space, and (ii) the ability to sample new datapoints from p(x) through p(z)p θ (x|z). These competing goals and objectives form a fundamental trade-off, and as a consequence, there is no easy way to measure the success of an auto-encoder model. Instead, measuring success requires careful consideration of multiple different metrics. The discussion of the metrics is in Section 2.2, and they will be incorporated in the IVELS tool (Section 5.1 and 5.2).For generating discrete sequences while controlling user-specific attributes, for example peptide sequences with specific functionality, it is crucial to consider conditional generation. The most Figure 1: Overview of the IVELS tool. In every stage, we can filter the models to select the ones with satisfactory performance. In the first stage, models can be compared using the static metrics that are typically computed during training (left). In the second stage, we investigate the activity vs noise of the learned latent space (top right) and evaluate whether we can linearly separate attributes (not shown). During the third stage, the tool enables interactive exploration of the attributes in a 2D projection of the latent space (bottom right). straightforward approach would be limiting the training set to those sequences with the desired attributes. However, this would require large quantities of data labeled with exactly those attributes, which is often not available. Moreover, the usage of those models that are trained on a specific set of labeled data will likely be restricted to that domain. In contrast, unlabeled sequence data is often freely available. Therefore, a reasonable approach for model training is to train a VAE on a large corpus without requiring attribute labels, then leveraging the structure in the latent space for conditional generation based on attributes which are introduced post-hoc. As a prerequisite for this goal, we focus on how q φ (z|x) encodes the data with specific attributes. We introduce the encoding of the data subset corresponding to a specific attribute, i.e. the subset marginal posterior, in Section 3. This will be important in the IVELS tool (Section 5.3 and 5.4). Now that we introduced our models (VAE family), the importance of conditioning on attributes, and our case study of interest (peptide generation), we turn to the focus of our paper. To assist in the model selection process, we present a visual tool for interactive exploration and selection of auto-encoder models. Instead of selecting models by one single unified metric, the tool enables a machine learning practitioner to interactively compare different models, visualize several metrics of interest, and explore the latent space of the encoder. This exploration is building around distributions in the latent space of data subsets, where the subsets are defined by the attributes of interest. We will quantify whether a linear classifier can discriminate attributes in the latent space, and enable visual exploration of the attributes with 2D projections. The setup allows the definition of new ad-hoc attributes and sets to assist users in understanding the learned latent space. The tool is described in Section 5.In Section 6, we discuss some observations we made using IVELS as it relates to our specific domain of peptide modeling and different variations of VAE models. We approach the unsupervised representation learning problem using auto-encoders (AE) BID13. This class of methods map an input x to continuous latent space z ∈ R D from which the input has to be reconstructed. Regular AEs can learn representations that lead to high reconstruction accuracy when they are trained with sparsity constraints, but they are not suitable for sampling new data points from their latent z-space. On the other hand, variational auto-encoders (VAE) BID16, which frame an auto-encoder in a probabilistic formalism that constrains the expressivity of z, allow for easy sampling. In VAE, each sample defines an encoding distribution q φ (z|x) and for each sample, this encoder distribution is constrained to be close to a simple prior distrbution p(z). We consider the case of the encoder specifying a diagonal gaussian distribution only, i.e. q φ (z|x) = N (z; µ(x), Σ(x)) with Σ(x) = diag[exp(log(σ 2 d)(x)))]. The encoder neural network produces the log variances log(σ 2 d)(x). The standard VAE objective is defined as follows (where D KL is the Kullback-Leibler divergence), DISPLAYFORM0 We explore a number of popular model variations we explored for the problem of modeling peptide sequences. With the standard VAE, we observe the same posterior collapse as detailed for natural language in BID3, meaning q(z|x) ≈ p(z) such that no meaningful information is encoded in z space. To address this issue, we introduce a multiplier β on the weight of the second KL term BID12 i.e. β-VAE with β < 1. BID0 analyze the representation vs. reconstruction trade-offs with a rate-distortion (RD) curve which also provides a justification for tuning β to achieve a different trade-off along the RD curve. We also considered two major variations in the VAE family: Wasserstein auto-encoders (WAE) BID29 and adversarial auto-encoders (AAE) BID21. WAE factors an optimal transport plan through the encoder-decoder pair, on the constraint that marginal posterior q φ (z) = E x∼p(x) q φ (z|x) equals a prior distribution, i.e. q φ (z) = p(z). This is relaxed to an objective similar to L VAE above, but with the per-sample D KL regularizer replaced by a divergence regularizing a whole minibatch as an approximation of q φ (z). In WAE training with maximum mean discrepancy (MMD) or with a discriminator (=AAE), we found a benefit of regularizing the encoder variance as in BID24 BID1. For MMD, we used a random features approximation of the Gaussian kernel BID22.In terms of model architectures, the default setting is a bidirectional GRU encoder and a GRU decoder. Skip-connections can be introduced between the latent code z and decoder output BID7, which was motivated by avoiding latent variable collapse. Alternatively, one can replace the standard recurrent decoder with a deconvolutional decoder BID30, which makes the decoder non-autoregressive, thus forcing it to rely more on z. The starting point of any model evaluation are the typical numerical metrics logged during training. Here we consider the following metrics:• Reconstruction log likelihood log p θ (x|z) and D KL (q(z|x)|p(z)), computed over the heldout set.• MMD between q φ (z) (average over heldout encodings q(z|x)) and the prior p(z).• Encoder log(σ 2 j (x)), averaged over heldout samples and over components j = 1... D. Large negative indicates the encoder collapsed to deterministic.• Reconstruction BLEU score on heldout samples.• Perplexity evaluated by an external language model for samples from prior z ∼ p(z) or from heldout encodings z ∼ q φ (z|x).All of our sample-based metrics (BLEU, perplexity) are using beam search as the decoding scheme. Metric of interest 2 Activity and encoder variance. We use (unit) activity as a proxy to investigate how many latent dimensions are encoding useful information about the input. We will extend the concept to evaluate whether the marginal posterior q φ (z) is far from the prior. Active units are defined as the number of dimensions d = 1... D where the activity DISPLAYFORM0 is above a threshold BID4. The activity tells us whether the encoder mean µ φ (x) varies over observations. To expand on this notion, we follow BID18 and focus on the marginal posterior q φ (z). In the usual parametrization where the encoder is used to specify the mean µ φ (x) and diagonal covariance Σ φ (x) of a Gaussian distribution, the total covariance is given by: DISPLAYFORM1 This tells us the d-th diagonal element of the covariance matrix Cov DISPLAYFORM2 is the sum of the activity A d and the average encoder variance σ 2 d (x) (i.e. how much noise is injected along this dimension on average). To satisify q φ (z) = p(z) we need at least the first and second order moments of q φ (z) to match those of p(z), and thus we need Cov q(z) [z] ≈ I. Inspecting the two terms of Cov q φ (z) [z] along the diagonal can thus tell us (i) an obvious way to violate closeness to the prior and (ii) whether the covariance is dominated by activity or encoder uncertainty. In this work, we focus on unconditionally trained VAE models. Even though several approaches have been proposed to incorporate attribute or label information during VAE training BID17 BID26, they require all labels to be present at training time, or require a strategy to deal with missing labels to enable semi-supervised learning. To avoid this, we follow BID8 and aim to train a sequence auto-encoder model unconditionally and rely on the structure of the latent z-space to define the attributes post-hoc. This process also eliminates the need for retraining the VAE when new labels are acquired or new attributes are defined. Specifically, from the interactivity standpoint, this enables end users or model trainers to interactively specify new attribute groups or subsets of the data. We aim to enable attribute-conditioned generation by understanding the learned latent space of a model. Let the attributes be a = [a 1, . . . a n], with each attribute a i taking a value y ∈ A i (typically: positive, negative, not available). Since the probability of the intersection of n attributes can be small or zero, we focus on conditioning on a single attribute at a time. In general, we define a subset S of our dataset as those datapoints where attribute a i = y and denote the corresponding distribution as p S (x) or p ai=y (x) = p(x|a i = y). By focusing on the subset S defined by selecting on a i = y, we have the flexibility to define new subsets online with the same notation. In the auto-encoder formulation and variants we discussed, an important object is the maginal posterior q φ (z) = E x∼p(x) q φ (z|x), which is introduced as the aggregate posterior by BID21. This distribution is central to the analysis of BID14 as well as WAE which relies on q φ (z) = p(z).Let us now define the marginal posterior for subset S with distribution p S (x): DISPLAYFORM0 The subset marginal posterior is an essential distribution for our analysis as it tells us how the distribution corresponding to a specific attribute is encoded in the latent space. Since we aim to be able to sample from q S φ (z) conditionally, we require the distribution to have two properties. First, q S φ (z) needs to be distinct from the distribution q φ (z) (Aim 1). We found that the underlying data-generating distributions of labeled and unlabeled data do not necessarily match for biological applications. Since there might be an underlying reason why a data point has not been labeled, a model should learn which points are of interest for a particular attribute. The second aim is that q S φ (z) should have sufficient regularity to capture the distribution with simple density modeling (Aim 2). Being able to discriminate between different attribute labels within the same category is crucial when aiming to generate sequences with a particular property. To be able to analyze arbitrary subsets in an interactive exploration of q S φ (z), we focus on the following two metrics of interest. This metric addresses the question of whether we can easily discriminate the subset S (corresponding to attribute a i) in the latent space. To address the aims introduced above, we consider two approaches to define S and S:• a i available vs a i not available (Aim 1).• a i = y vs a i = y two different attribute labels (e.g., y =positive, y =negative) (Aim 2).Metric of interest 4 2D projections of the full marginal posterior q φ (z) and the place of each subset marginal posterior q S φ (z) in it. While static metrics can provide an intuition for the quality of the latent space, we further aim to analyze the well-formedness of the space. Thus, we investigate how attributes cluster visually in 2D projections for different models. Peptides are single linear chains of amino acids. We consider sequences that are composed of twenty natural amino acids, i.e., our vocabulary consists of 20 different characters. The length of sequences is restricted to ≤ 25. Depending on the amino acid composition and distribution, peptides can have a range of biological functions, e.g., antimicrobial, anticancer, hormone, and are therefore useful in therapeutic and material applications. Latent variable models such as VAEs have been successfully applied for capturing higher-order, context-dependent constraints in biological peptide sequences BID23, for semisupervised generation of antimicrobial peptide sequences BID6, and for revealing distinct cancer-specific methylation patterns of DNA sequences BID28. BID10 have used VAE models for automatic design of chemicals from SMILES strings. In this study, we focus on comparing the latent spaces learned by modeling the peptide sequences at the character level by the use of VAE and its variants. Furthermore, we investigate the feasibility of using the latent space learned using a large corpus of unlabeled sequences to track representative distinct functional families of peptides. For this purpose, we mainly focus on five biological functions or attributes of peptides, i.e. antimicrobial (AMP), anticancer, hormonal, toxic and antihypertensive. Frequently, these peptides are naturally expressed in a variety of host organisms and therefore are identified as the most promising alternative to conventional small molecule drugs. As an example, given the global emergence of multidrug-resistant bacterias or "superbugs" and a dry discovery pipeline of new antibiotics, AMPs are considered as exciting candidates for future infection treatment strategies. Water solubility is also considered as an additional attribute. Our labeled dataset comprises sequences with different attributes curated from a number of publicly available databases BID25 BID9 BID15 BID2 BID11. Below we provide details of the labeled dataset: The labeled data are augmented with the unlabeled dataset from Uniprot-SwissProt and UniprotTrembl BID5, totaling ∼ 1.7 M points. The data is split into separate train, valid, and test sets corresponding to 80%, 10% and 10% of the data, respectively. Data for which an attribute is present is upsampled with a factor 20× during training. DISPLAYFORM0 Our tool aims to support the role of a model trainer as described by BID27. This role does not assume extensive domain knowledge, but an understanding of the model itself. As such, we DISPLAYFORM0 limit the visual elements of the tool to those that do not evaluate sequences at a granular peptide-level. As a consequence, the tool aims to visualize high-level information that is captured during training and iteratively focuses down to the medium-level, where we evaluate attributes. Specifically, we introduce a three-level pipeline that enables a user to conduct a cohort-analysis of all the models they have trained. During each level, the user can filter the remaining models based on information provided by the tool. The iterative filtering process further allows successively more computationally expensive analyses. The subsections here follow the measures highlighted as "Metric of interest" in the previous sections. The details of the models appearing in the figures are found in Appendix A. The first level FIG2 is a side by side view of a user-specified subset of the metrics logged during training (rows) across multiple models (columns). In our example, we select only the model checkpoints from the final epoch of each training run. These metrics would be typically inspected by a model trainer in graphs over time, for example in tensorboard. However, while graphs over time excel at communicating for a single metric, comparing models across multiple metrics is challenging and time-consuming. By showing an arbitrary number of metrics at once, the IVELS tool enables the selection of promising models for further analysis. The tool further simplifies the selection process through the ability to sort the columns by different metrics. Sorting makes it easy to select models that achieve at least a certain performance threshold. FIG3 presents the encoding of Metric of Interest 2, i.e. the diagonal of Cov q φ (z) [z]. For interpretability, we sort the latent dimensions according to decreasing activity. This visual representation allows inspection of the balance between activity (encoder mean changing depending on observations), and average encoder variance (how much the noise is dominating). We discuss the observations for different models in Section 6.2. Figure 4: Level 2.2. Attribute discriminability in the latent space. Despite being trained fully unsupervised, all models successfully encode multiple attributes in z. Given that we established that z is actively encoding information in the first part of level 2, the second part of level 2 aims to evaluate whether we can linearly separate attributes within the learned space. This is a prerequisite for level 3, which assumes that the encodings can be related to the attributes in a meaningful way. Figure 4 presents the for the models across the attributes that are available in the dataset. Following the Metric of Interest 3, we differentiate between positive and negative labels (indicated by lab) and labeled and unlabeled samples (indicated by between). For each of these scenarios, we train a logistic regression on z of the training set and evaluate it on the training, validation and test sets. To account for a dynamic number of different labels, the for lab are the accuracy, whereas between is measured in AUC. The y axis is scaled in. We allow the user to select either t-SNE BID19 or linear projection on axes of interest (PCA). To visualize the different attributes, we enable color-coding in two modes: show labels and compare labels. " Show labels" will color-code the different values a single attribute can assume, in our case positive and negative. " Compare labels" allows to select two different attributes with a specific value. Using this mode, we can for example examine whether there is a section of the latent space that has learned both soluble and AMP peptides. Should a data point have been annotated with both of the selected values, it is color-coded with a separate color. 6 DISCUSSION AND From stage 2 (Fig. 4) it is encouraging to see that in this unconditionally trained model the different attributes are salient as judged by a simple linear classifier trained on the latent dimensions. Some general trends appear. We can observe generally high performance across all models, which is promising for conditional sampling. One difference of note is that the β-VAE and AAE models perform worse on the AMP attribute. The discriminators for attributes with limited annotation, specifically water-solubility, overfit on the training set which indicates a need for further annotation efforts. The further demonstrate that toxicity is more challenging than the remaining attributes despite being the second-most common attribute in the dataset. These set the stage for further investigation of the latent space. Fig. 5 shows the tSNE projections of the latent space learned using three different models (Level 3). Two distinct attributes, positive antimicrobial (amp_amp_pos, in blue) and antihypertensive (antihyper_antihyper, in red), are also shown. Interestingly, these two attributes are more well-separated in the latent space obtained using AAE and WAE (Fig. 5, middle and right) compared to that from VAE (Fig. 5, left). From the biological perspective, antihypertensive peptides are distinct in terms of length, amino acid composition, and mode of action. Antimicrobial peptides are typically longer than the antihypertensive ones. Also, antimicrobial peptides function by disrupting the cell membrane, while antihypertension properties originate from enzyme inhibition BID20. From the Cov q φ (z) [z] plot (Level 2.1, FIG3, we can observe that the VAE (column 3) suffers from posterior collapse, since it has no active units. We can further see that the β-VAE (column 1 and 2) address the collapse issue and about half of its dimensions in z space are active. Interestingly, the dimensions that are not active become dominated by encoder variance, such that the total variance for each dimension is close to 1. The skip-connection added to the GRU (2nd column) lead to a slightly higher activity around the tail of the active dimensions, though the difference is minor. The WAE and AAE (column 4 and 5) have relatively little encoder variance, meaning they are almost deterministic. Notably, the WAE covariance is furthest away from the prior. From the t-SNE plots (Fig. 5) we see the WAE and AAE show good clustering of the attributes like positive antimicrobial and antihypertensive, showing that the latent space is clearly capturing those attributes, even though they were not incorporated at training time. We presented a tool for Interactive Visual Exploration of Latent Space (IVELS) for model selection focused on auto-encoder models for peptide sequences. Even though we present the tool with this use case, the principle is generally useful for models which do not have a single metric to compare and evaluate. With some adaptation to the model and metrics, this tool could be extended to evaluate other latent variable models, either for sequences or images, speech synthesis models, etc. In all those scenarios, having a usable, visual and interactive tool for model architects and model trainers will enable efficient exploration and selection of different model variations. The from this evaluation can further guide the generation of samples with the desired attribute(s).

represent the short summary for finding the corresponding scientific article (long)

We present a visual tool to interactively explore the latent space of an auto-encoder for peptide sequences and their attributes.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.