A Multiversion Programming Inspired Approach to Detecting Audio Adversarial Examples
Qiang Zeng Jianhai Su Chenglong Fu Golam Kayas Lannan Luo















A Multiversion Programming Inspired Approach to Detecting Audio Adversarial Examples^⋆
    1st Given Name Surname
dept. name of organization (of Aff.) 

name of organization (of Aff.)

City, Country 

email address
2nd Given Name Surname
dept. name of organization (of Aff.) 

name of organization (of Aff.)

City, Country 

email address
3rd Given Name Surname
dept. name of organization (of Aff.) 

name of organization (of Aff.)

City, Country 

email address
4th Given Name Surname
dept. name of organization (of Aff.) 

name of organization (of Aff.)

City, Country 

email address
5th Given Name Surname
dept. name of organization (of Aff.) 

name of organization (of Aff.)

City, Country 

email address
6th Given Name Surname
dept. name of organization (of Aff.) 

name of organization (of Aff.)

City, Country 

email address

    December 30, 2023
============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================




plain
plain


Adversarial examples (AEs) are crafted by adding
human-imperceptible perturbations to inputs such that a 
machine-learning based classifier incorrectly labels them.
They have become a severe threat to the trustworthiness of machine learning.
While AEs in the image domain have been well studied, 
audio AEs are less investigated. Recently,
multiple techniques are proposed to generate audio AEs,
which makes countermeasures against them urgent.
Our experiments show that, given an audio AE, 
the transcription results by 
Automatic Speech Recognition (ASR) systems differ significantly (that is,
poor transferability),
as different ASR systems use different architectures, parameters,
and training datasets. 
Based on this fact and inspired by Multiversion Programming,
we propose a novel audio AE detection approach MVP-Ears, which utilizes the diverse
off-the-shelf ASRs to determine whether an audio is an AE. 
We build the largest audio AE dataset to our knowledge, and
the evaluation shows that the detection accuracy reaches 99.88%.

While transferable audio AEs are difficult to generate at this moment,
they may become a reality in future. 
We further adapt the idea above to proactively train the detection system
for coping with transferable audio AEs. Thus, the proactive detection 
system is one giant step ahead of attackers working on transferable AEs.







Adversarial Example, transferability, Automatic Speech Recognition, DNN.




§ INTRODUCTION

Automatic Speech Recognition (ASR) is a system that converts speech to text. ASR has been studied intensively for decades. Many various technologies, such as gaussian mixture models and hidden Markov models, were developed. In particular, recent advances <cit.> based on deep neural networks (DNNs) have improved the accuracy significantly. DNN-based speech recognition 
thus has become the mainstream technique in ASR systems. 
The industrial companies including Google, Apple and Amazon
have widely adopted DNN-based ASRs for interacting with 
 IoT devices, smart phones and cars. Gartner <cit.> estimates that, by 2020, 75% of American households will have at least one smart voice-enabled speaker, where DNN-based ASR plays a
critical role.



Despite the great accuracy improvement,
recent studies <cit.> show that DNN is vulnerable to adversarial examples. An adversarial example (AE) x' is a mix of a host sample x and a carefully crafted, human-imperceptible perturbation δ such that a DNN will assign different labels to x' and x. Figure <ref> shows an audio AE example, which sounds as in the text shown on the left to human, but is transcribed by an ASR system into a completely different text as shown on the right.



Several techniques 
have been proposed to generate audio AEs.


There are two state-of-the-art audio AE generation methods:
(1) White-box attacks:
Carlini et al. propose an optimization based method to convert
an audio to an AE that transcribes to an attacker-designated phrase <cit.>. It is classified as white-box attacks because the target system's detailed internal architecture and parameters are required to perform the attack.

(2) Black-box attacks:
Taori et al. <cit.> combine the
genetic algorithm <cit.> and gradient estimation to 
generate AEs. This attack method does not require the knowledge of 
the ASR's internal parameters, but imposes a larger perturbation (94.6% similarity on average 
between an AE and its original audio, compared to 99.9% in  <cit.>).
The rapid development of audio AE generation methods makes countermeasures against audio AEs an urgent and important problem. 






The goal of our work is to detect audio AEs.





Existing work on audio AE detection is rare and limited.
Yang et al. <cit.> hypothesized that AEs are fragile:
given an audio AE, if it is cut into two sections, which are then transcribed separately,
then the transcription by splicing the two sectional results is very different from the
result if the AE is transcribed as a whole. 
However, as admitted by the authors <cit.>, this method cannot handle “adaptive attacks”,
which may evade the detection by embedding a malicious command into one section alone.
Rajaratnam et al. <cit.> proposed detection based on
audio pre-processing methods. Yet, if an attacker knows the detection details,
he can take the pre-processing effect into account when generating AEs. 
Such attacks have been well demonstrated for bypassing
similar techniques for detecting image AEs <cit.>.
An effective and robust audio AE detection method  is missing.

Our key observation is that existing ASRs are diverse in terms of architectures, parameters and training
processes. Our hypothesis is that an AE that is effective on one ASR system is likely 
to fail on another, which is verified by our experiments.
How to generate transferable audio AEs that can fool multiple heterogeneous ASR systems is still 
an “open question” <cit.> (discussed in Section <ref>).
This inspires us to borrow the idea of multiversion programming (MVP) <cit.>,
a method in software engineering where multiple functionally equivalent programs are
independently developed from the same specification, such that an exploit that
compromises one program is ineffective on others.
We thus propose to run multiple ASR systems in parallel, and an input is determined
as an AE if the ASR systems generate very dissimilar transcription results. 

Moreover, while it is unknown how to systematically generate transferable audio AEs at this moment,
we predict that such techniques may be proposed in future. We thus aim to handle
transferable audio AEs as well, which is a notable challenge to our system for
two reasons. First, since there are no transferable audio AEs, how can our 
machine learning based AE detector be trained? Second, since such hypothetical transferable AEs
can fool multiple ASRs, how can this idea still work?

Our first insight is that our detector essentially is not trained using
AEs, but similarity scores (which are calculated based on the similarity of transcription results
of different ASRs). Thus, if we assign a high similarity score between two ASRs, it simulates
the effect that an AE can fool both. This way, we can conveniently generate
a dataset of hypothetical AEs in the form of vectors of similarity scores. 
Our another insight is that, due to the complexity and diversity of ASRs,
it is difficult, if not impossible, to generate audio AEs to fool all ASRs
in foreseeable future. In light of this, we generate a dataset
of hypothetical AEs that are rather transferable but cannot fool
all ASRs. We make use of this dataset to proactively
train an audio AE detection system that can keep resilient 
to transferable AEs, as long as there is still one ASR that
cannot be fooled by the AEs. The code, datasets and models are publicly
available.[<https://github.com/quz105/MVP-audio-AE-detector>]

We made the following contributions.


    
  * We empirically investigate the transferability of audio AEs
with multiple experiments
and analyze the reasons behind the poor transferability.


  * To our knowledge, we build the largest audio AE dataset, which
can be reused by interested researchers. 


  * We propose
a novel audio AE detection approach, called MVP-Ears, inspired by multiversion programming, which
reaches accuracy 99.88%.
The detection method dramatically
reduces the flexibility of the adversary, in that audio AEs cannot succeed 
unless the host text is highly similar to the malicious command.


  * We propose the idea of proactively training a transferable-AE detection system,
such that our system is one giant step ahead of attackers who are working on generating transferable AEs.




The rest of this paper is organized as follows. We provide some background knowledge about the ASR system's general architecture and audio AE generation in Section <ref>. Then, we discuss the transferability of audio AEs in Section <ref>. We describe the main idea and system architecture in Section <ref> and present the detailed evaluation in Section <ref>. Section<ref> gives a survey about related works. Finally, we present some discussion in Section <ref> and conclude in Section <ref>.



§ BACKGROUND



 §.§ Automatic Speech Recognition System



An Automatic speech recognition (ASR) system is used to automatically interpret human's speech audio into texts <cit.>.
ASR has been an active research topic since the first digit recognizer Audrey was published by Bell Laboratories in 1952.
  As illustrated in Figure <ref>, the process of converting an audio into a text sentence typically involves 
the following four stages: feature extraction, acoustic feature recognition, phoneme assembling, and language generation.

The input is an audio represented in the form of a waveform. 
(1) Feature Extraction. The input audio is first segmented into short frames, each of which is converted into feature vectors, 
such as MFCC feature vectors, LPC feature vectors, and PLP feature vectors. 
Because MFCC approximates the human auditory system's response more closely than others, MFCC feature vector is considered as the most
suitable frequency transformation format for speech data, and thus 
adopted by most recent ASR systems <cit.>. 
(2) Acoustic feature recognition. The extracted feature vectors are then recognized by the acoustic model as phonemes; the
phoneme is the minimal unit of sounds of languages. 

(3) Phoneme assembling. Next, the combined phonemes are used to estimate the potential phoneme letter sequence, where the dictionary model is used to correct the word spelling of the phoneme letter sequence.
(4) Language generation. At the last step, the generated words
are further adjusted to the contexts and merged to the final text sentence by the language model. 


The core part of an ASR system is the acoustic feature recognition stage, which outputs phonemes. A phoneme is affected by the sound in the corresponding frame and also the two adjacent frames. Different ASR models utilize different approaches to recognizing
phonemes. 
As the features are processed with certain correlations, temporal 
models are well utilized, e.g.,a hybrid system of GMM and HMM (GMM-HMM). GMM-HMM is used as a statistical classifier to convert the feature vectors into phonemes, but it is limited in dealing with overlapping sound from different sources <cit.>. Recently,
DNN-HMM <cit.> 
is widely used to 
analyze the features, and can provide better recognition performance. 

However, when training GMM-HMM and DNN-HMM based ASR systems, each audio sample needs to be aligned and labeled manually at the phoneme level in order to assemble phonemes into words, which is time
consuming and error-prone. To address this issue, Connectionist Temporal Classification based on Recurrent Neural Network (RNN-CTC) <cit.>  is proposed and used for end-to-end speech recognition. CTC can directly convert the feature vectors into words without the need of generating phonemes as the intermediate result. 
Thus, CTC-based ASR systems outperform HMM-based ones.

Deepspeech <cit.> is the most popular open-source implementation of a CTC-based ASR system, and is maintained regularly by Mozilla. As a result, Deepspeech is often used as the attack target model by many recent audio AE generation works.




 §.§ Audio Adversarial Examples


Despite the great advances and diverse applications of machine learning and deep learning <cit.>,
adversarial examples (AEs) <cit.> can be crafted to fool them. As explained by Goodfellow et al. <cit.>, the existence of adversarial examples is due to the neural network's linear nature. Based on this observation, they propose the Fast Gradient Sign Method (FGSM) as a simple but efficient algorithm to generate AEs.





Although a lot of successful AE attacks targeting image classification systems have been realized <cit.>, this is not the case for 
the audio domain. 
Unlike DNN-based image classifiers where the pixels of an image are directly used as inputs of DNN, 
an audio needs to be first segmented into short frames with each converted into feature vectors, which are then fed into the DNN model of the ASR. Moreover, different ASRs use different lengths to segment
an audio. These make the audio AE generation much more challenging. For example, by directly applying FGSM, Cisse et al. can only generate adversarial spectrogram instead of an audio AE <cit.>. 




White-box AEs.
Carlini et al. overcome this limitation by adding the MFCC reconstruction layer into the backpropagation optimization of gradients and provide an end-to-end method for creating targeted audio AEs with the white-box setting <cit.>. It is classified as white-box attacks because the target ASR system's detailed internal architecture and parameters are required to generate AEs.







Black-box AEs.
With a black-box setting, the targeted ASR internal architecture and parameters are not known.
Alzantot et al. <cit.> make the first effort to generate black-box AEs. They adopt the genetic algorithm to iteratively add noises into an input audio sample and discard outputs with bad performance in each generation. The iteration ends up with a black-box AE that can fool a single-word classification system (not ASR). 
Taori et al. <cit.> extend this work and combine the genetic algorithm with gradient estimation to 
generate black-box AEs, which can fool Deepspeech.
However, the genetic algorithm-based method imposes a larger perturbation (94.6% similarity on average 
between a black-box AE and its original audio, compared to 99.9% between a white-box AE and its original audio). 
Their current AE generation only embeds up to two words in one audio. 












the  audio is first segmented into short frames with a sliding window. Then, acoustic features extraction is applied to each frame toconvert the frame into a vector with the pre-defined dimension. Commonly used acoustic feature extraction techniques include Mel-scale Frequency Cepstral Coefficients (MFCC), Linear Predictive Coefficient (LPC), and Perceptual Linear Predictive (PLP). MFCC is the most commonly used acoustic feature extraction technique by most recent ASR systems <cit.>. The extracted vectors are classified by the acoustic model as phonemes <cit.> which are units of sound. With the help of the words dictionary and language model, the phonemes are assembled into words and sentences.

The core part of the ASR system is the phoneme recognition. A phoneme is not only related to the sound in the corresponding frame but also affected by two adjacent frames. The general solution is representing each phoneme with a three-state Hidden Markov Model (HMM) and construct words by concatenating a sequence of phoneme-level HMMs. As the classifier used to convert the frame features into phonemes, Gaussian Mixture Model (GMM) is first used as a statistical classifier (GMM-HMM) but is limited on dealing with overlapping sound from different sources. In recent research works <cit.> DNN is widely used combined with the HMM (DNN-HMM) to provide better recognition performance. 

Although powered by the most advanced Deep Learning technique, DNN-HMM approaches only replace the GMM with DNN and still follow the old ASR framework that requires three separated models (the acoustic model, the dictionary, and the language model). This cast two drawbacks: 1) the three models are trained independently with the training target of each component different from the whole ASR system's target (low word error rate) which means the combination of three models may not be optimal though each part is optimal; 2) the training audio need to be labeled manually in phoneme level which causes heavy workload. To address these problems, Connectionist Temporal Classification (CTC) <cit.> is used for end-to-end speech recognition. CTC directly converts the features into characters without producing the phoneme sequence as the intermediate result. CTC-based ASR systems surpass the DNN-HMM based approaches by automatically aligning the input text transcripts with all neural network implementations.  Deepspeech <cit.> is the most popular open source implementation of CTC-based end-to-end ASR systems that is  maintained and iteratively updated by Mozilla. Deepspeech is also chosen as the attacking target by many recent audio AE generation works.






N-version programming (NVP) is first introduced in 1977 <cit.> as a method to enhance the reliability of software and computer systems. This technique is  independently developing  multiple functionally equivalent versions of software from the  same specification. On deployment, multiple software is executed concurrently and on the same task. At some checkpoints, executing instances generate comparison vectors to check the consistency of the execution. After that, different instances reach consensus on the execution states and proceed to the next stage. The most significant benefit of the NVP is relaxing the rigorous requirement of the reliability of software by providing fault tolerance from the system level. Since the multiple version of software are developed independently, the probability of they share the identical flaws are very small. Especially, some implementation specific flaws usually only occur in one  instances. The software flaws of any instance not only affect the execution flow but also cause inconsistency among comparison vectors. On detecting inconsistency, some decision algorithms are applied to determine the correct execution flow and prevent the crashing of the execution. The concept has already been used as an effective defense method against software flaws <cit.> and are widely used in development of highly reliable software such as  flight control software on modern airliners. Aside from the fault tolerance, it's also proved to be effective to defend attacks that exploit 0-day software vulnerabilities <cit.>.



 §.§ Audio adversarial examples

Adversarial examples (AEs), which is first discovered by Szegedy et al. <cit.>, are sophisticatedly crafted input samples to fool artificial intelligence systems. As explained by Goodfellow et al. in <cit.>, the existence of adversarial examples are caused by the neural network's linear nature. Based on this observation, they propose the Fast Gradient Sign Method (FGSM) as a simple but efficient algorithm to reliably generate adversarial examples. Following this method, a lot of successful attacks targeting image classification systems are realized <cit.>. In FGSM, given the loss function of the target neural network and the related parameters, FGSM linearize the loss function and generate an " optimal max-norm constrained perturbation"  of

    η = ϵ sign ( ∇ _  x  J ( θ ,  x  , y ) )
 where the J ( θ ,  x  , y ) is the original loss function and the θ is the model's parameter. By iteratively applying the method, classification output can be manipulated to move towards the expected target.

Unlike the pixel level input for DNN-based image classifier, pro, the input waveform is pre-processed by sliding window segmentation and MFCC feature extraction into a spectrogram, which is then passed into the DNN. This makes the audio AE generation more challenging. By directly apply the FGSM, Cisse et al.'s work <cit.> is only capable to generate adversarial spectrogram instead of the raw audio file. Carlini et al. are the first to overcome this limitation by adding the MFCC reconstruction layer into the backpropagation optimization of gradients and provide an end-to-end method to create targeted audio AEs with the whitebox setting.

With blackbox setting, targeting DNN's loss function is no longer accessible. Alzantot et al. make the first effort to utilize the genetic algorithm to iteratively apply noise to the input audio sample and discard outputs with bad performance in each generation. The iteration ends up with a version that can successfully fool a simple command classification system. Taori et al. extend the work in <cit.> to successfully generate targeted AEs that can fool the deepspeech ASR system.
To be mentioned, the genetic algorithm based approaches are much slower than those FGSM whitebox attacks.



 Following this method, a lot of successful attacks targeting image classification systems are realized <cit.>. In FGSM, given the loss function of the target neural network and the related parameters, FGSM linearize the loss function and generate an " optimal max-norm constrained perturbation"  of

    η = ϵ sign ( ∇ _  x  J ( θ ,  x  , y ) )
 where the J ( θ ,  x  , y ) is the original loss function and the θ is the model's parameter. By iteratively applying the method, classification output can be manipulated to move towards the expected target.

Unlike the pixel level input for DNN-based image classifier, pro, the input waveform is pre-processed by sliding window segmentation and MFCC feature extraction into a spectrogram, which is then passed into the DNN. This makes the audio AE generation more challenging. By directly apply the FGSM, Cisse et al.'s work <cit.> is only capable to generate adversarial spectrogram instead of the raw audio file. Carlini et al. are the first to overcome this limitation by adding the MFCC reconstruction layer into the backpropagation optimization of gradients and provide an end-to-end method to create targeted audio AEs with the whitebox setting.

With blackbox setting, targeting DNN's loss function is no longer accessible. Alzantot et al. make the first effort to utilize the genetic algorithm to iteratively apply noise to the input audio sample and discard outputs with bad performance in each generation. The iteration ends up with a version that can successfully fool a simple command classification system. Taori et al. extend the work in <cit.> to successfully generate targeted AEs that can fool the deepspeech ASR system.
To be mentioned, the genetic algorithm based approaches are much slower than those FGSM whitebox attacks.










§ TRANSFERABILITY OF AES



There are two types of adversarial examples (AEs): non-targeted AEs and targeted AEs. 
A non-targeted AE is considered successful as long as it is classified as a incorrect label, while a targeted AE 
is successful only if it is classified as a label desired by attackers. 

In the context of audio based human-computer interaction, a non-targeted AE is not very useful, as it cannot
make the ASR system issue an attacker-desired command. This explains why state-of-the-art 
audio AE generation methods all generate targeted AEs <cit.>. 
Thus, our work focuses on detection of targeted AEs,
although the proposed approach is also effective in detecting non-targeted AEs (see Section <ref>). 




 §.§ Transferability in Image Domain


An intriguing property of AEs in the image domain is the existence of transferable adversarial examples. That is, an image AE crafted to mislead a specific model can also fool another different model <cit.>. By exploiting this property, Papernot et al. <cit.> propose a reservoir-sampling based approach to generate transferable non-targeted image AEs and successfully launch black-box attacks against both image classification systems from Amazon and Google. 

But a recent study <cit.> points out that the transferability property does not hold in some scenarios: their experiment shows a failure of AE's transferability between a linear model and a quadratic model.

For targeted image AEs, Liu et al. <cit.> 
propose an ensemble-based approach to craft AEs that can transfer to ResNet, VGG and GoogleNet 
models with success rates of 40%, 24%, and 11%, respectively.
Thus, generating transferable image AEs is a resolved problem <cit.>.




 §.§ Transferability in Audio Domain: Open Question
 


To the best of our knowledge, no systematic approaches have been proposed to generate
transferable audio AEs. Carlini 
et al. <cit.> find that the attacking method derived from Fast Gradient Sign Method <cit.> in the image domain is ineffective for generating audio AEs because of a large degree of non-linearity in ASRs, and further state that the transferability of audio AEs is an open question.


A recent work CommanderSong <cit.> aims to generate AEs that can fool an ASR system in the presences of background noise. 
This work also slightly explored transferability of audio AEs (see Section 5.3 of the paper <cit.>). Specifically, to create AEs that can transfer from
Kaldi to DeepSpeech (both Kaldi and DeepSpeech are open source ASR systems, 
and Kaldi is the target ASR system of CommanderSong), a two-iteration recursive AE generation
method is described in CommanderSong: an AE generated by CommanderSong, embedding a malicious 
command c and able to fool Kaldi, is used as a host sample in the
second iteration of AE generation using the method <cit.>, 
which targets DeepSpeech v0.1.0 and embeds the same command c. 

We followed this two-iteration recursive AE generation method to generate AEs, but our experiment results <cit.> showed that the generated AEs
could only fool DeepSpeech but not Kaldi. That is, AEs generated using this method 
are not transferable. 




Furthermore, we adapted the two-iteration AE generation method by concatenating the two aforementioned 
state-of-the-art attack methods <cit.> 
and <cit.> targeting DeepSpeech v0.1.0 and v0.1.1, respectively, 
expecting to generate AEs that can fool both DeepSpeech v0.1.0 and v0.1.1. But none of the generated AEs 
showed transferability <cit.>. 

Moreover, by changing the value of “" from 1 to 3, which is a parameter configuration of the Kaldi model, we derived a variant of Kaldi.
The AEs generated by CommanderSong did not show transferability on the variant,
even given the fact that the variant
was only slightly modified from the model targeted by CommanderSong. 
Here, we clarify that CommanderSong did not claim their AEs
could transfer across the Kaldi variants. 




Based on our detailed literature review and empirical study, we find that so far
there are no systematic  methods that 
can generate transferable audio AEs effective across diverse ASR systems. 
This is consistent with the statement by Carlini 
et al. <cit.> that transferability of audio AEs is an open question.


























In a most recent work <cit.>, Xu et al. claim they can craft tranferable audio AEs by manipulating the low-level features of the input audio example. However, as admitted by the author, it is only untargeted attack which is not useful for attackers, and is beyond the scope of our work as discussed at the beginning of this section.

Except the aforementioned paper,  <cit.> is the only work claims that their AE generation method CommanderSong—which
uses a method similar to <cit.>—can craft AEs that transfer from Kaldi 
(their attack target system) to iFlytek. We contact the authors and receive 10 CommanderSong AEs.

We test all the 10 AEs on iFlytek. But our experiment  results (recorded as videos <cit.>) show that none successfully fooled
iFlytek. Moreover, by changing the value of “" from 1 to 3, which is a parameter configuration of the Kaldi  ASR system used in CommanderSong, we derive a variant of Kaldi. However, none of the received samples shows transferability 
on the variant either, even given the fact that the variant
is only slightly modified from the model targeted by CommanderSong. 

The authors of CommanderSong describe an interesting way to generate AEs which they claim can fool both
Kaldi and DeepSpeech <cit.>:
an AE generated by CommanderSong—which has a command embedded and targets Kaldi—is further used as the host sample to generate an AE that embeds the same command but targets DeepSpeech. The authors claim that the resulting AE can fool both Kaldi and DeepSeech. This audio AE generation approach seems to be promising to craft transferable audio AEs. However, based on our experiments (Section <ref>), the attacking effect of two iterations cannot be simply superposed. As a result, the generated AEs can only attack the ASR targeted at the second iteration.


Based on our detailed literature review and empirical study (Section <ref>), we find that so far
there are no systematic AE generation methods that 
can generate transferable audio AEs effective across two different ASR systems, not to mention three or more. 
This is consistent with the statement by Carlini 
et al. <cit.> that transferability of audio AEs is an open question.














§ MVP-INSPIRED AUDIO AE DETECTION




 §.§ Multi-Version Programming


Multi-version programming (MVP) was first introduced in 1977 to enhance the reliability of software and computer systems <cit.>. The main idea of MVP is to independently develop multiple programs based on the  same specification. At runtime, multiple programs are executed concurrently and perform the same task. At each checkpoint, each program generates the result, which is to check the consistency of the execution. After that, all programs reach consensus on the execution states, and then proceed to the next stage. 

The most significant benefit of MVP is relaxing the rigorous requirement of the reliability of software by providing fault tolerance from the system level. Since the multiple software programs are developed independently, the probability that they share the same flaw is very small. Especially, some implementation specific flaws usually only occur in one program. The software flaws of any program not only affect the execution flow but also cause inconsistency among comparison results. Upon detecting inconsistency, some decision algorithms are applied to determine the correct execution flow and prevent the crashing of the execution. Such a concept has already been used as an effective defense method against software flaws <cit.>, and are widely used in development of highly reliable software, such as flight control software on modern airliners. Beside the fault tolerance, it has also been proved to be effective to detect attacks that exploit zero-day software vulnerabilities <cit.>.








 §.§ MVP-Inspired Idea
 
Given a system with MVP,
an exploit that compromises one program probably fails on other programs. 
This inspires us to propose a system design that runs multiple ASRs in parallel. 
The intuition behind this design is that different ASRs 

can be regarded as ”independently developed programs” in MVP.
Since they follow the same specification—that is, to covert audios into texts.
Given a benign sample, they should output very similar recognition results.
On the other hand, an audio AE can be regarded as an ”exploit”, and cannot fool all ASR systems
as illustrated in Section <ref>.
Thus, by comparing the results of the multiple ASRs, we are able to determine
whether an audio is an AE or not.



This idea is comparable with the ensemble approach, where multiple 
detection methods are combined to form a stronger one <cit.>. But they do not adopt a
concise architecture like ours (e.g., <cit.> requires different
input processing methods, while <cit.> needs
a generalist and multiple specialists), which simply runs
multiple ASRs in parallel for detection. Our work is in spirit 
similar to an independent work <cit.>. However, they differ
in the following aspects: (1) That work <cit.> aims to detect image AEs, while our work detects audio AEs. More critically,
their approach uses the softmax layer outputs as features
for AE detection and attackers can thus adaptively generate the AE that leads to similar 
softmax outputs between models, while we use the final transcription 
outputs for AE detection and adaptive attacks cannot succeed unless transcriptions
are similar, which is difficult as discussed in Section <ref>.
(2) That work only considers the bi-model design, while we consider a more general N-model design. 
(3) We do not stop at detecting
existing audio AEs, but propose the idea for proactively training systems to detect 
transferable audio AEs.






 §.§ Architecture








Figure <ref> shows the system architecture.
It consists of a target ASR, multiple auxiliary ASRs, 
a similarity calculation component, and a binary classifier. 
The target ASR is the model targeted by the adversary (e.g., the speech 
recognition system at a smart home), and each auxiliary ASR is 
a model that is different from the target ASR. The detection of AEs 
involves the following three steps.




  * An audio is first fed into the target ASR and auxiliary ASRs. Each
ASR independently and simultaneously converts the audio 
into a text sentence (i.e., a transcription). 


  * The transcriptions are then sent to the similarity calculation component,
which calculates similarity scores between the transcription recognized by the target ASR and
that by each auxiliary ASR. 


  * Finally, these similarity scores are passed into a binary classifier
to determine whether the audio is adversarial.







The second step, similarity calculation, involves the following two sub-steps. 



  * Each transcription is converted into its phonetic-encoding representation. Phonetic encoding converts a word to the representation of its pronunciation <cit.>. This helps handle variations between ASRs, as they may output different words
for similar sounds. The validity of using phonetic encoding is demonstrated in Section <ref>. 


  * For each auxiliary ASR, a similarity score is calculated
to measure the similarity between the transcriptions generated by the 
target ASR and the auxiliary ASR. 
We have tried different similarity measurement methods, and finally
adopted the Jaro-Winkler distance method <cit.>
due to the higher detection accuracy
(see Section <ref>). It outputs a score ∈ [0,1], where 0 indicates very dissimilar and 1 similar.









 §.§ Diverse ASRs


Yu and Li <cit.> summarized the recent progress in deep-learning based ASR acoustic models, where both recurrent neural network (RNN) and convolutional neural network (CNN) come to play as parts of deep neural networks. Standard RNN could capture sequence history in its internal states, but can only be effective for short-range sequence due to its intrinsic issue of exploding and vanishing gradients. This issue is resolved by the introduction of long short-term memory (LSTM) RNN <cit.>, which outperforms RNNs on a variety of ASR tasks <cit.>. 

As to CNNs, its inherent translational invariability facilitates the exploitation of variable-length contextual information in the input speech. The first CNN model proposed for ASRs is the time delay neural network <cit.> that applies multiple CNN layers. Later, several studies <cit.> combine CNN and Hidden-Markov Model to create hybrid models that are more robust against vocal-tract-length variability between different speakers.
However, currently, there is no single uniform structure used across all ASRs.

In addition to several opensource systems, many companies have independently 
developed their own proprietary ASRs, such as Google Now, Apple Siri, and Microsoft Cortana. 




In our system, we use the following three ASRs, including DeepSpeech, Google Cloud Speech, and Amazon Transcribe. 
DeepSpeech is an end-to-end speech recognition software opensourced by Mozilla based on Baidu's research paper <cit.>. 
DeepSpeech v0.1.0 <cit.> uses a five-layer neural network where the fourth layer is a RNN layer,
while DeepSpeech v0.1.1 follows the same architecture and makes some improvements on the implementation. Even though these two ASR systems are very similar, our experiments (Section <ref>)
show  that, when the two are used to build an MVP-inspired
detection system, 
the AE detection accuracy is still very high. This means that for any target ASR system, we have potentially many candidate auxiliary ASR systems available 
without having to worry about the extent of (dis)similarity of the models.



Unlike DeepSpeech, the DNN behind Google Cloud Speech is a LSTM-based RNN according to the source <cit.>. Each memory block in Google's LSTM network <cit.> is a standard LSTM memory block with an added recurrent projection layer. This design enables the model's memory to be increased independently from the output layer and recurrent connections. 

However, for Amazon Transcribe, there is no available public information about its internal
details. 

In short, existing ASR systems are diverse with regard to architectures, parameters and training datasets. 
Compared to opensource systems, proprietary ASR systems provide little information that can be exploited by attackers. 
Given the diversity of ASR systems (and proprietary networks), it is unclear how
to propose a generic AE generation method that can simultaneously mislead all of them. 




Target ASR.
Our system uses DeepSpeech v0.1.0 as the target ASR. The reason is the opensource white-box AE generation
method <cit.> targets DeepSpeech v0.1.0.
Note that the generation of white-box AEs
requires the knowledge of the model architecture and parameters.



Auxiliary ASRs.
We use DeepSpeech v0.1.1, Google Cloud Speech, and
Amazon Transcribe as the auxiliary ASRs. 
They are all off-the-shelf widely used ASRs.


























Table <ref> shows a typical example, where the AE can only fool one ASR but fails on others. It illustrates that,
given an AE, the recognition result by the target ASR differs significantly from those by the auxiliary ASRs.



 §.§ Questions to Be Explored

There are still
several system design questions to be answered.
(Q1) How to measure
the similarity of two transcriptions?
(Q2) How many auxiliary ASRs work the best?
More ASRs make the system more robust
and resilient to attacks, but will they introduce more false positives?
(Q3) Which classification algorithm works best?
(Q4) Transferable audio AEs cannot be created
systematically at this moment, but they
may become a reality in future. How can our
system be proactively trained to deal with them?


We choose to perform experiments to guide and validate
the system design and answer
these questions in Section <ref>.


 


§ EXPERIMENT-GUIDED SYSTEM DESIGN AND THE EVALUATION

We finalize some system design details according to experiment results and
evaluate the accuracy and robustness of the system. 
We first describe the experiment settings (Section <ref>)
and discuss the dataset used in our evaluation (Section <ref>). 
We then investigate the feasibility of our idea (Section <ref>), and examine different methods for calculating
similarity scores and select the best one (Section <ref>).
Next, we evaluate the accuracy of our system when
one auxiliary ASR is used (Section <ref>) and more than one auxiliary 
ASR is used (Section <ref>), 
and also evaluate the robustness of our system 
against unseen attack methods (Section <ref>).
After that, we conduct an experiment to examine whether our system can work well when facing  (hypothetical) transferable AEs  (Section <ref>).
We next evaluate the time overhead due to detection (Section <ref>).
Finally, we evaluate the detection effectiveness against
non-targeted AEs (Section <ref>).






 §.§ Experimental Settings





Our experiments were performed on a 64-bit Linux machine with 
an Intel i9-7980XE CPU @ 2.60GHz (18-core), dual NVIDIA GeForce GTX 1080 Ti, 
and 32GB DDR4-RAM.	

As explained in Section <ref>, we 
use DeepSpeech v0.1.0 <cit.> (called ) as the target model. 
 DeepSpeech v0.1.1 <cit.> (called ), Google Cloud Speech <cit.> (called ), and Amazon Transcribe <cit.> (called ) are the three auxiliary ASRs. 

We use +{, ..., } to
denote a system using  as the target model and , ...,  as the auxiliary models.

For example, +{} means
a system using  as the target model and  as the single
auxiliary model. 








	














 §.§ Dataset Preparation
 

We consider two audio AE generation techniques:
white-box based <cit.> and 
black-box based <cit.> methods,

and build two datasets: 
a Benign dataset and an AE 
dataset, each of which contains 2400 audio samples, as shown in 
Table <ref>.
The audio samples of the Benign dataset are randomly selected from the dev_clean dataset of LibriSpeech <cit.>.

The AE dataset consists of the following two parts. (1)
1800 white-box AEs, including 990 AEs provided by <cit.> and 810 created by us. We created the AEs following the style of <cit.> by using randomly selected samples from
LibriSpeech as the host audios to embed the English sentences provided by <cit.>. (2) 600 black-box AEs constructed by applying the black-box approach <cit.>;
each AE embeds only two words. The two-word limit is due to the current capacity of <cit.>.
Each white-box AE takes 18 minutes on average to generate, while each black-box
AE takes 90 minutes. The datasets and the transcription details are made
publicly available.[<https://goo.gl/CJmrQh>]
We have verified that
all AEs can successfully fool the target model .






















 §.§ Feasibility Analysis
 

To validate the feasibility of the MVP-inspired idea, we 
feed all the benign samples and AEs into each of the four ASRs,
and calculate the similarity scores. Given an audio (benign sample or AE),
the similarity score indicates the similarity of the transcription of  and
that of an auxiliary ASR.

The histograms in Figure <ref> confirm
the feasibility of the idea: in each of the three cases, 
the similarity scores for AEs and those for benign samples 
form two almost disjoint clusters. It means that benign samples
lead to relatively high similarity scores, while AEs low scores.

 














 §.§ Choosing Similarity Calculation Methods
 


Many methods can be used to measure the similarity between two strings.
Some well-known ones include
Jaccard index <cit.>, 
Cosine similarity, 
and Edit distance (e.g., JaroWinkler <cit.>).
In addition, we propose to convert the transcription into
its phonetic encoding (PE) <cit.>. 
Our hypothesis is that this may help reduce the false positives:
given a benign sample, different ASRs may output different words of
similar pronunciations, which can lead to  higher similarity scores
after phonetic encoding.

To choose the similarity measurement method and validate the
hypothesis above, we consider six different combinations of
similarity calculation methods, as shown in
Table <ref>.
For example, PE_JaroWinkler means phonetic encoding and
JaroWinkler are used as the similarity calculation method. 
In each case, we train a SVM based classifier 
using 80% of audios in the two datasets (Benign and AE), and then
test the classifier using the rest 20% (that is, 4800 * 0.2 = 960 audios). 
Table <ref>
shows that the combination PE_JaroWinkler
achieves the highest accuracies in all the four example systems.
For example, when , ,  are
used as the auxiliary ASRs, it achieves the accuracy 99.90%(,
although other methods also lead to high accuracies at least 99.58%).
Thus, we choose PE_JaroWinkler 
as the similarity calculation method. 













	


	
		
		













 §.§ Effectiveness of Single-Auxiliary-Model Systems
 

A detection system that uses a single auxiliary ASR 
has the advantage of a cheap deployment. We build three such systems:
+{}, +{}, and 
+{}. 
We use k-fold cross validation (k = 5) to evaluate the three systems:
the datasets are divided into 5 equal subsets and, in each of the five runs,
one subset is used for testing and the rest four for training. 
As shown in Table <ref>, the mean and standard deviation of the results
across the 5 runs are reported.

We use three different binary classifiers, including ,  and , and configure each classifier as follows: (1)  uses a 3-degree polynomial kernel; (2)  uses 10 neighbors to vote; and (3)  uses a seed of 200 as the starting random state.

Based on the results, we conclude that (1) all the single-auxiliary-model systems
achieve high accuracies (over 98%) and low FPRs/FNRs; and (2) 
SVM performs slightly better than the other two classifier methods.

But it is worth mentioning that if the auxiliary ASR (like Kaldi) 
is not accurate in recognizing benign audios, the AE detection accuracy is
bad (e.g., 80% with Kaldi).
















 §.§ Effectiveness of Multi-Auxiliary-Model Systems
 
A multi-auxiliary-model system uses 
more than one auxiliary model. We 
build four multiple-auxiliary-model systems, denoted as:
+{, },
+{, },
+{, },
and +{, , }.




For a multi-auxiliary-model system with n auxiliary models, 
given an input audio, n similarity 
scores are calculated and form a feature vector. 
The feature vector is then fed into the binary 
classifier to predict whether the audio is an AE. 

Table <ref> shows the testing results using 5-fold cross validation. All the accuracy results are
higher than 99.70%, and FPR and FNR are lower than 0.30%, regardless of 
the auxiliary models and binary classifiers used. 
The three-auxiliary-model system performs the best and reaches the accuracy
99.88%. We also observe that a system with more auxiliary models achieves
better accuracy, probably because extra models provide more features in 
the feature vector. 



Does FPR increase due to more auxiliary ASRs?
To answer this intriguing question, we extract the FPR and
FNR results when SVM is used as the binary classifier from Table <ref> and Table <ref> and obtain Table <ref>,
which shows that both FPRs and FNRs tend to slightly decline when more
auxiliary ASRs are used. This conclusion holds when either 
KNN or RandomForest is used as the binary classifier.















 §.§ Robustness against Unseen Attack Methods
 

This experiment aims to examine whether a system trained on AEs generated by a particular attack method is able to detect AEs generated by other kinds of attack methods—such an AE is called an unseen-attack AE.  

We use the defense rate, defined as the ratio of the number of successfully 
detected AEs among the total number of AEs, to measure
the robustness. 




Single-auxiliary-model systems.
We first examine all the three single-auxiliary-model systems.
We train each system using only the 2400 benign samples, and test it  
using all the 2400 AEs (see Table <ref>),
which all can can be considered as unseen-attack AEs.


We simply use a similarity score threshold T for detecting AEs:
an audio that has a score lower than T is detected as an AE.
First, the threshold is determined by having the FPR less than 5%.
The detection results are presented in 
Table <ref>.

Each of the three detection systems achieves excellent defense rates 
≥ 99.83%. 
Second, we vary T and obtain the Receiver Operating Characteristic (ROC) 
curve, as shown in Figure <ref>.
The AUC in each case is close to 1, implying high detection accuracies,
which is consistent with the results of the machine learning based approaches (see Section <ref>).










Multiple-auxiliary-model systems.
We next examine the four multiple-auxiliary-model systems. 
As aforementioned, two different methods 
are used to generate AEs:
the white-box approach and black-box approach. 
We have totally 1800 white-box AEs and 600 black-box AEs.

We conduct two different experiments to evaluate each of the four  
multiple-auxiliary-model systems.

We first use all the 1800 white-box AEs and 1800 benign samples to train 
each system, and use all the black-box AEs to test each trained system. 
The results are showed in the second column in Table <ref>.
We can see that all the systems perform very well, and the lowest defense 
rate is 99.17% for the +{} system.

We next use all the 600 black-box AEs and 600 benign samples to train each system, and use all the white-box AEs to test each trained system. 
The results are showed in the third column in Table <ref>.
There are three systems achieve the defense rate of 100%—that is,
all the white-box AEs can be successfully detected. For the system  +{, }, it achieve a high defense 
rate of 99.89%.

Therefore, we can conclude that our detection systems are very robust against
unseen-attack AEs. 
































 §.§ Detecting Hypothetical Multiple-ASR-Effective AEs


In this experiment, we seek to understand whether our proposed system can work well when facing AEs that can fool more than one ASRs, which we call multiple-ASR-effective (MAE) AEs. 

The problem here is the lack of methods for generating such MAE AEs. We thus propose to create hypothetical MAE AEs. Note that we do not really create a real MAE AE in the form of an audio, instead we synthesize a feature vector to represent it.
In the following presentation, we refer to a hypothetical MAE AE as a MAE AE. 






Creating MAE AEs.
We use a multiple-auxiliary-model system, +{, , }, in this experiment. Thus, for a given input audio, 
its feature vector contains three similarity scores.  
We have the following critical observation.
If an MAE AE successfully fools the target model and an auxiliary model, 
both models should convert the audio into transcriptions the same as or highly
similar to the 
command desired by the attacker (since it is a targeted attack). 
This AE works just like a benign sample (whose transcription is the command)
for the perspective of the two models. Thus, the similarity
score of the AE corresponding to the auxiliary model should be 
as high as that of a benign sample. We thus construct the feature
vectors for MAE AEs as follows. 

First, from the previous experiments, we collect two pools of similarity scores: one contains the similarity scores for the 2400 benign samples, denoted as λ_Be; and the other contains the similarity scores for the 2400 AEs, denoted as λ_Ak. 





Second, to create an MAE AE that can fool  and , for example, we randomly select a similarity score from λ_Be (representing that this AE can successfully attack  and ), and two similarity scores from λ_Ak (representing that this AE cannot attack  and , resulting in a low similarity score for each ASR). The created MAE AE is denoted as 
AE(,). 

Similarly, consider creating an MAE AE that can fool , , and , as
another example. This AE is denoted as 
AE(,,).
Two similarity scores are selected from λ_Be (representing that this AE can successfully attack , , and ), and one similarity score from λ_Ak (representing that this AE cannot attack ). 





Through this, we create six different types of MAE AEs, as listed in Table <ref>. Each type contains 2400 MAE AEs. 





Accuracy.
For each type of MAE AEs, we construct six datasets:
each dataset contains 2400 benign samples and 2400 the corresponding MAE AEs.
For each dataset, 80% of its benign samples and MAE AEs are used for training, and the remaning 20% for testing. 

We use SVM as the binary classifier, and PE_JaroWinkler to measure the similarity. 





 
 
Table <ref> shows the testing results. We can see that the systems trained on different types of MAE AEs have very high accuracies (higher than 97%), and low FPRs and FNRs.  









Robustness to unseen-attack MAE AEs.
We further investigate whether the system is able to detect unseen-attack MAE AEs.  
We also include the original AEs in this experiment. Specifically, we use one type of AEs 
and the benign samples to train the system, and use another type of AEs to test 
the trained system. The result is presented in Table <ref>.

We can conclude that (1) all the systems trained on different types of AEs work very well on the original AEs—achieving more than 99% defense rates (the second column in Table <ref>).

(2) If a system is trained on one type of MAE AEs that can fool a set of ASRs, Λ={A_1,...,A_n} where A_i is an ASR, then it can defend against another type of MAE AEs that can fool a set of the ASRs, Λ^'⊆Λ, with almost 100% defense rates (the italic numbers in Table <ref>). For example, if the system is trained on Type-4 MAE AEs that can fool 
, , and , then it can defend against Type-1 MAE AEs that can fool  and .





(3) We especially notice that some defense rates are quite low: for example, if the system is trained on Type-2 MAE AEs that can fool  and , its defense rate against Type-5 MAE AEs that can fool ,  and  is only 16.04%. 
This is expected, as the training data has never seen Type-5 MAE AEs.















Comprehensive system.
To build a comprehensive system that can defend against all the 6 types of MAE AEs, we build a system using Type-4, Type-5 and Type-6 MAE AEs.
They constitute a dataset of totally 7200 MAE AEs, and we also build another dataset
of (feature vectors corresponding to) 7200 benign samples. 

We use 80% of them for training and the rest for testing.
The detection result on the testing set (of the three types of MAE AEs and benign samples) shows 97.22% detection accuracy, 3.47% FPR and  2.08% FNR. The system's 
defense rates over original AE, Type-1, Type-2 and Type-3 MAE AEs are impressive, as shown in Table <ref>. All the four types of AEs are 100% detected.


We thus conclude the MVP-inspired idea lets us proactively train systems that
can detect transferable AEs even before their existence. This is a big 
step ahead of attackers working on methods for generating transferable
audio AEs.



 §.§ Overhead Measurement
 
We evaluate the time overhead imposed by the detection.
Both Google Cloud Speech and Amazon Transcribe are remote cloud
based services, whose delays are dominated by networks and 
service dispatching. We are more interested in the overhead
imposed by the detection itself. Thus, we measure the time overhead
on +{}. We use SVM
as the binary classifier, and evaluate using all the
2400 AEs and 2400 benign samples in our datasets. 
The average recognition time of  
is 8.8 seconds per audio. The time overhead in +{}
comprises three parts: recognition overhead (due to the parallel ASRs), 
similarity calculation overhead and  classification overhead,
which are 0.065s (0.74%), 5.0e-06s and 4.2e-07s.
Thus, they incur negligible overheads. 






 §.§ Detecting Non-targeted Attacks


Non-targeted AEs make the ASR system output 
incorrect transcriptions. Compared to targeted AEs, 
attacks based on non-targeted AEs are much weaker, as they cannot
fool the ASR system to generate attacker-desired results. 







We find that non-targeted AEs actually can be generated by simply adding noises to 
benign audio samples. We randomly choose 118 benign audio samples from CommonVoice dataset <cit.> which are labeled with US accent and high-quality vote.
 The noise is added to the audio samples with SNR set to -6dB. After adding the noise, the output sample is still recognizable to humans but the ASR transcription  results have higher than 80% word error rate (WER). 
 
We treat non-targeted AEs as unseen-attack AEs and use the same evaluation
method used in Section <ref>. Specifically, 
we choose the similarity score threshold by having the FPR as 5%,
and use that threshold to determine whether an input is an AE.
The testing results show that the defense rate is higher than 90%
no matter which auxiliary ASR is used. The defense rate is
lower than the case of targeted attack AEs, mainly
because of the  relatively small WER when generating non-targeted AEs.








































 §.§ Empirical Study of Transferability of Audio AEs


Our last experiment is an empirical study of the transferability of audio AEs.
As CommanderSong describes a method to generate AEs that they claim can fool both Kaldi and DeepSpeech <cit.>, we first contact the authors to get ten CommanderSong AEs that can attack Kaldi. Among the ten AEs, two AEs can already attack both Kaldi and , and three AEs cannot be used for the transferability experiment. For the remaining five AEs, we follow their method to generate two types of transferable AEs: 

(1) For each CommanderSong AE, we apply the white-box attack to embed the same command as the original AE, and generate a new AE that targets . Our expectation is that this new AE can fool both Kaldi and , denoted as AE → [Kaldi,]. We generate five 
AEs → [Kaldi,].

(2) For each CommanderSong AE, we first apply either the white-box or black-box attack to generate an intermediate AE targeting , which are then applied on the black-box attack to generate a new AE targeting . Our expectation is that this new AE can fool at least  and , denoted as AE → [,]. We also generate five such AEs.

For each AE, we feed it to each of its two target ASRs to detect whether the AE can successfully fool both two ASRs. The results show that all the AEs generated by us can only fool the ASR that is targeted at the last iteration.
For example, AE → [,] 
can only fool . 
For the two CommanderSong AEs provided by the CommanderSong authors,
they achieve the transferability and can attack both Kaldi and . However, the two AEs contain a significant amount of noise that even make the original song unclear to hear and can be easily perceived by human ears. This is also pointed out in <cit.>. Therefore, the attacking effect of two iterations cannot be simply superposed.

Based on our investigation, we find that, to transfer an AE targeting 
one ASR for attacking another ASR, a lot of perturbations are required to be inserted into the AE, and thus a significant amount of noise will be introduced to the host audio; on the other hand, this would be easily noticed by human. Therefore, so far there is no AE generation method that can successfully generate transferable audio AEs that can be used in practice.













































To evaluate the effect of the concatenating attack as proposed in <cit.>, we conduct two  experiments. First, We contact the authors to get some CommanderSong (dentoed as ) (word-to-api) AE audio samples which are targeting the Kaldi (denoted as ) ASR system.  We follow their two-stage concatenating AE generation by applying Whitebox attack <cit.> (denoted as ) targeting deepspeech version 0.1.0 (denoted as ) on received Commandersong  AEs to embed the same target text. Then, we adopt the two-stage recursive AE generation method by concatenating the whitebox attack <cit.> and blackbox attack <cit.> which are targeting DeepSpeech v0.1.0 and v0.1.1 respectively, expecting to generate AEs that can fool both DeepSpeech v0.1.0 and v0.1.1. We try all combinations of two attack methods and two target ASR systems to embed the same content for two stages. Finally, for each outputted AE sample, we use its two target ASR systems to transcribe it and calculate their phonetic similarity scores as illustrated in Section <ref>.  

The evaluation results are listed in Table <ref>. CSW1∼CSW5 are for the first  experiment that have Kaldi and Deepspeech v0.1.0 as the target ASR systems. Transcripts of the stage 1 ASR system are obviously messed up while those of the stage 2 ASR system remain high fidelity. Aside from these five + AEs crafted by our own, we also evaluate the two AE samples (CS1,CS2) provided by CommanderSong's authors with the same technique and find the two examples do achieve transferability across  Kaldi and Deepspeech. However, all these + AEs contain a significant amount of noise that even overshadows original song in the host audio and can be easily perceived by human ear. This is also pointed out in <cit.>.  For the second type of experiment, we have WW1∼WW5, BB1∼BB5, WB1∼WB5, and BW1∼5 for all four combinations.  The transcriptions of the first stage target ASR system are unreadable, which indicates the effect of the first stage attack is destroyed by the following stage attack. As for the similarity score, most samples have very low phonetic similarity scores  0.75. From the overall evaluation results of the concatenating attack, we can safely claim that it's not an effective method to generate transferable AEs, even for two very similar ASR systems (deepspeech v0.1.0 and v0.1.1). 

Beyond this conclusion, we also get another observation that audio AE generation methods have different levels of robustness in face of further perturbations and more robust generation method will introduce more noises to the host audio and vice versa. The commanderSong that has a chance to generate transferable AEs for Kaldi and Deepspeech causes the most significant perturbation (15dB of perturbation,90% similarity between the AE and host) which can be easily perceived by human. The blackbox attack gets lower scores but has a higher similarity of 94.6% . The Whitebox attack which is the most vulnerable to the second stage perturbation (WB1∼WB5) with lowest average score achieves the highest fidelity of %99.95 similar and -31dB perturbation. BAsed this observed pattern, we propose the hypothesis that for a possibly successful concatenating AE generation attack, the stage 1 attack must be much more robust to withstand the distortion caused by the stage 2 attack Which means more amount of noise introduced. However, as shown in our experiment result, even by introducing annoying noise, CommanderSong is still not robust enough to go through the perturbation by the whitebox attack which adds the least noise. As a result, even some future AE generation methods can be successfully concatenated to fool two ASR systems, the produced transferable samples will carry obvious distortion which greatly undermines the power of the so-called "hidden command attack" and concatenation of more than two stages to fool more ASR systems would cause even more distortion. 











In the last experiment, we are interested in the proposed system's performance when it is attacked by AEs that can fool more-than-one ASRs. Due to the lack of existing methods for creating multiple-ASRs-effective (MAE) AEs, we decide to create hypothetical AEs for the experiment. We do not create an actual audio for a hypothetical AE, but synthesize a feature vector to represent it. Each element of the feature vector is the similarity score of the hypothetical AE's transcriptions recognized by the target ASR and one auxiliary ASR used in our system design. For a MAE AE, its transcriptions recognized by its targeting ASRs should be similar and thus the corresponding scores should be high. Based on this intuition, taking a 2-ASRs-effective AE for example, which can fool  and , the AE's similarity score with regard to  and  will be randomly selected from the pool of 2400 benign samples' corresponding similarity scores. While the AE's similarity score with regard to  and another ASR (not ) will be randomly selected from the pool of real 2400 AEs' corresponding similarity scores.

To make the discussion convenient, we use {} to denote an AE that could attack ASRs in the set, {}. Thus, the real 2400 AEs are labeled as {}. Since DS0 is the victim targeted by both of the whitebox and blackbox methods <cit.> and additioanl three ASRs are availabel to us, 6 types of MAE AEs are investigated in this experiment, which are listed in Table <ref>. 

For each kind of these AEs (including the real AEs), we use them together with the 2400 benign samples to build a detection system using SVM as the binary classifier, PE_JaroWinkler as the similarity metric and all the three ASRs as the auxiliary models. The detection system built upon {} is detenoted as _{}. The performance of all detection systems towards their test sets is reported in Table <ref>. As seen all detection systems built on 2ME hypothetical AEs have a detection accuracy of higher than 99%, a 0% FNR and a decent FRP of no more than 3.75%; while for detection systems constructed upon 3ME hypothetical AEs, the performance drops slightly, but still hold a detction accuracy of higher than 96%, low FPR and FNR of no more than 5.34% and 2.5% correspondingly.


We further test each built detection system with unseen hypothetical AEs. Here, unseen hypothetical AEs are the hypothetical AEs that are not used in building the detection system. The result is reported in Table <ref>. _{} is the system built with actually generated AEs. One interesting observation we find is that Detection_AEASR1, ASR2, ..., ASRn could effectively identify AEs that can attack a subset of ASR1, ASR2, ..., ASRn.

























	



















Multiple detection systems based on the proposed system design were tested against two targeted audio AE generation techniques, the white-box approach <cit.> and the black-box one <cit.>. 
DeepSpeech v0.1.0 (DS0) is picked as the target model since both target DS0 <cit.>. 
While auxiliary models are Google Cloud Speech <cit.> (GCS), Amazon Transcribe <cit.> (AT) and DeepSpeech v0.1.1 <cit.> (DS1). The first two ASRs are on-line services, while the third one runs locally with an officially pre-trained model. Tested detection systems are divided into two groups, single-auxiliary-model (SAM) systems and multiple-auxiliary-models (MAM) ones. To further evaluate the robustness of our system design, we did an experiment that launches attacks of unknown AEs that do not participate in the establishment of a testing detection system.







and (Part 3) 50 “stress-test” AEs used to test the extreme cases, i.e.,
when the embedded text of an AE is only two or three words different
from the corresponding host speech's transcription. 


For example, if the transcription of a host speech is “just one thing more”, one possible transcription of a stress-test AE 
is “just order more”. 

The host speeches are selected from the
Common Voice dataset. To generate a stress-test AE, we obtain a intended command by applying two or three operations to the transcription of a host speech. One operation may insert, delete or replace a word. Then, we use the white-box approach <cit.> to convert the host speech into an AE such that the AE can be transcribed as the intended command by the target ASR.





The reason we especially consider the AEs of Part 3 is to test
whether our proposed system work well on the extreme cases: when the transcriptions of an AE and its corresponding host speech are very similar (only one or two words differ). For such extreme cases, we 
seek to understand whether the AEs can bypass the detection of our system.




























For each multiple-auxiliary-models system, we train a binary classifier on all the 
3840 feature vectors (1920 benign samples and 1920 AEs), and test it on over 960 test samples. 
We use three different binary classifiers, including , 
 and , 

and configure each classifier as follows: (1)  uses a 
3-degree polynomial kernel; (2)  uses 10 neighbors to vote;
and (3)  uses a seed of 200 as the starting random state. 








Figure <ref> shows the ROC results for the three single-auxiliary-model systems.










For a given input audio, a single-auxiliary-model system computes a similarity score 
between two transcriptions, one generated by the target model  and another
by the auxiliary model.
If the similarity score is lower than a threshold T, the input
audio is detected as adversarial, otherwise benign.
We use the Receiver Operating Characteristic (ROC) 
curve to select the threshold T, such that the sum of TPR (true positive rate) and TNR (true negative rate) is maximized. 


Figure <ref> shows the ROC results for the three single-auxiliary-model systems.
And the training results (including TPR and TNR) are shown in Table <ref>.


For a given input audio, a single-auxiliary-model system computes a similarity score 
between two transcriptions, one generated by the target model  and another
by the auxiliary model. We show the mean and standard deviation of each testing performance in Table <ref>. All of the single-auxiliary-model systems achieve detection accuracy higher than 98% and FPR (false positive rate) lower than 
2.04%. We further investigate the results, and have an 
interesting finding: for one FP case, the transcriptions generated 
by  and  have the same meaning, but are in 
different formats. The transcription by  is the phrase
“SEVENTEEN SEVENTEEN”, while that by  is 
the digit “1717”. The reason is that the alphabet model of 
 does not include digits; as a result, any digit is represented as a word. But if the alphabet model is integrated into ,
such a FP case could be eliminated.


For both types of Single-Auxiliary-Model Systems (<ref>) and Multiple-Auxiliary-Models Systems (<ref>), we run 5-fold validation and report the mean and standard deviation of interested performances. In each run of the 5-fold validation for each system, we train a binary classifier on all the 3840 feature vectors (1920 benign samples and 1920 AEs), and test it on over 960 test samples. We use three different binary classifiers, including ,  and , and configure each classifier as follows: (1)  uses a 3-degree polynomial kernel; (2)  uses 10 neighbors to vote; and (3)  uses a seed of 200 as the starting random state.





§ RELATED WORK



 §.§ Audio Adversarial Example Generation


An AE generation method can be categorized as whitebox/blackbox and targeted/non-targeted.


Tavish et al. <cit.> and Carlini et al. <cit.> proposed methods for generating commands that are recognized 
by ASRs but are considered as noise by humans. However, the produced sound examples are noises and incomprehensible to humans, which greatly undermines the power of their attacks. 

To address this limitation, <cit.> proposed a semi-targeted AE generation method to embed text to host audios with similar content. 
Carlini proposed the first targeted audio AE generation system, but is is a whitebox method <cit.>.
Alzantot et al. proposed a blackbox scheme to generate  adversarial audio examples targeting  a simple speech command classification model (not an ASR) <cit.>.
Taori et al. <cit.> combined the
genetic algorithm <cit.> and gradient estimation to 
generate blackbox AEs. Finally, Yuan et al. aimed to craft audio AEs that remain effective when they are played over the air<cit.>. 






 §.§ Audio Adversarial Example Defense and Detection


The emergence of adversarial examples has attracted researchers to study its defense strategies. Many studies on detecting image AEs have been reported, such as <cit.>, while only a few are presented to cope with audio AEs, probably because techniques for crafting audio AEs just surfaced in the past two years. This also makes countermeasures against audio AEs urgent and important. 




Rajaratnam et al. <cit.> proposed to detect audio AEs based on
audio pre-processing methods. Yet, if an attacker knows the detection details,
he can take the pre-processing effect into account when generating AEs. 
Such attacks have been well demonstrated for bypassing
similar techniques for detecting image AEs <cit.>.
<cit.> further pointed out that the input transformation only gives a false sense of robustness against AEs by imposing obfuscation over gradients which can be circumvented by their proposed method. 



Carlini et al. <cit.> trained a logistic regression classifier, which was trained using a mix of benign and Hidden-Voice-Command (HVC) audios. But it can only detect hidden voice commands,
 instead of general audio AEs.





Yang et al. <cit.> proposed to identify audio AEs based on the assumption that audio AEs need complete audio information to resolve temporal dependencies. They cut the input audio into two sections, which were then transcribed separately. If the input is an AE, the result obtained by splicing of the two sectional results will be very different from the result when the input is transcribed as a whole. However, as admitted by the authors, their method cannot handle “adaptive attacks”, which evade the detection by only embedding malicious commands into one section alone. In short, the
literature has not reported an effective and robust audio AE detection method like ours.













§ DISCUSSION

If the malicious command embedded in an AE and the host transcription are very similar, our method will probably fail as their similarity score is high. But note that, prior to our work, the existing AE generation methods claim that any host audio can be used to embed a malicious command <cit.>. Our detection method dramatically reduces this attack flexibility, in that the attack cannot succeed unless the host transcription is similar to the malicious command. 


The current prototype system successfully demonstrates the 
feasibility of the Multiversion Programming
inspired approach to detecting audio AEs, and the novel idea of 
proactively preparing a detection system resilient to transferable AEs, which
may be generated in future.
However, to deploy such a system still needs more engineering efforts.
For example, the online Amazon Transcribe ASR imposes large delays to return the transcription result
immediately, probably because of the server load control on the Amazon side.
However, we argue that such delays are inherent in our idea or system design. 
The delays as well as the deployment barrier can be eliminated 
by running multiple high-quality ASRs locally. For example, by
using DeepSpeech v0.1.0 and DeepSpeech v0.1.1
to build the MVP-Ears system, it imposes negligible
delays (see Section <ref>) and achieves satisfactory accuracy
(99.56%). 



Although how to generate transferable AEs is still an open question currently, in future
it may become reality. We estimate our system should still be effective in detecting transferable AEs 
(unless the transferable AEs can fool all ASRs).
Our future work is to proactively prepare our detection system to detect them. 
The main difficulty is to get a training dataset of transferable AEs, which do not exist. However, a unique advantage of our design is that the binary classifier takes
similarity scores as inputs, rather than real AEs.
Thus, we can generate similarity scores that simulate the effect of
hypothesized highly-transferable AEs to train the model. 













§ CONCLUSION

Research on handling audio AEs is still limited. 
Considering that ASRs are widely deployed in smart homes, smart phones and
cars, how to detect audio AEs is an important problem. 
Inspired by Multiversion Programming, we propose to run multiple different ASR systems in parallel,
and an audio input is determined as adversarial if the multiple ASRs generate very dissimilar transcriptions.
Detection systems with one single auxiliary ASR achieve satisfactory accuracies (98%), while
systems with more than one ASR reach even higher accuracies (99.88%) as more features are provided to
the classifier. 
The research results invalidate the widely-believed claim that an adversary can embed a malicious command to any host audio. 

In addition, we propose the novel idea of training a proactive detection system for handling transferable audio AEs, such that the detection keeps effective as long as the AE is not able to
fool all the ASRs in the detection system. Therefore,
it makes our system one stride ahead of attackers working on generating transferable audio AEs.







§ ACKNOWLEDGMENT

This project was supported by NSF CNS-1815144 and NSF CNS-1856380.
The authors would like to thank Nicholas Carlini for sharing their AEs <cit.>,
and anonymous reviewers for their comments and suggestions.
In the early stage of this project, we used the Chameleon Cloud (funded by NSF),
so we would like to thank this project <cit.>.




IEEEtran
 

