title: DeepVOX: Discovering Features from Raw Audio for Speaker Recognition in Non-ideal Audio Signals
authors: Anurag Chowdhury Arun Ross
text: 
  
  
  	
  	
  	
  	
  	
  	
  	DeepVOX: Discovering Features from Raw Audio for Speaker Recognition in 
  		Non-ideal Audio Signals
      Anurag Chowdhury, Member, IEEE,
  		Arun Ross, Senior Member, IEEE
  		Both authors are with the Department
  			of Computer Science Engineering, Michigan State University, East Lansing,
  			MI, 48823, USA, 
  			e-mail: {chowdh51, rossarun}@msu.edu.
  	
      
  =======================================================================================================================================================================================================================================================
  
  
  		Automatic speaker recognition algorithms typically use pre-defined filterbanks, such as Mel-Frequency and Gammatone filterbanks, for characterizing speech audio. However, it has been observed that the features extracted using these filterbanks are not resilient to diverse audio degradations. In this work, we propose a deep learning-based technique to deduce the filterbank design from vast amounts of speech audio. The purpose of such a filterbank is to extract features robust to non-ideal audio conditions, such as degraded, short duration, and multi-lingual speech. To this effect, a 1D convolutional neural network is designed to learn a time-domain filterbank called DeepVOX directly from raw speech audio. Secondly, an adaptive triplet mining technique is developed to efficiently mine the data samples best suited to train the filterbank. Thirdly, a detailed ablation study of the DeepVOX filterbanks reveals the presence of both vocal source and vocal tract characteristics in the extracted features. Experimental results on VOXCeleb2, NIST SRE 2008, 2010 and 2018, and Fisher speech datasets demonstrate the efficacy of the DeepVOX features across a variety of degraded, short duration, and multi-lingual speech. The DeepVOX features also shown to improve the performance of existing speaker recognition algorithms, such as the xVector-PLDA and the iVector-PLDA.
  		
  	
  	
  	
  	
  	
  		Speaker Recognition, Degraded Audio, Deep Learning, Feature Extraction, Filterbanks
  	
  	
  	
  
  	
  
  § INTRODUCTION
  
  	
  	Automatic speaker recognition entails recognizing an individual from their voice. One of the key applications of speaker recognition is securing devices with voice-controlled user interfaces (VUI) such as digital voice assistants <cit.> and telephone banking systems <cit.>. VUIs are gaining popularity due to the ease-of-access provided by their hands-free operation. However, in practice, the voice input to the speaker recognition systems often exhibits non-ideal speech audio characteristics, such as degraded <cit.>, multi-lingual <cit.>, and short-duration <cit.> speech. The unfavorable nature of these non-ideal inputs propagates through different components of the speaker recognition system and lowers its performance <cit.>. Therefore, it is important to develop speaker recognition techniques that are robust to a wide variety of non-ideal audio conditions, thereby providing generalizable speaker recognition performance.
      
      Some of the current speaker recognition enabled consumer devices address the issues of non-ideal audio conditions at the sensor-level by employing specialized hardware, such as far-field microphone arrays <cit.>. However, the use of specialized hardware interfaces limits their backward compatibility with existing speaker recognition systems. On the other hand, some of the latest speaker recognition techniques address these issues at the software-level by designing robust matchers <cit.>. However, these techniques rely on traditional handcrafted speech features such as Mel-Frequency Cepstral Coefficients (MFCC) and Linear Predictive Coding (LPC), whose representation capability varies with the quality of input audio, thus limiting the effectiveness of the subsequent matcher <cit.>. While some of the recent work in end-to-end speaker recognition can perform speaker recognition directly from raw input audio, their robustness to non-ideal audio conditions is yet to be determined <cit.>.
  
      We position our work with the existing literature by approaching the issue of non-ideal audio conditions at the feature-level. We design a raw audio-based speech feature extractor, called DeepVOX, that is robust to non-ideal audio conditions and compatible with existing speaker recognition algorithms. Our method delivers generalizable noise-robust speaker recognition performance without any specialized hardware interface or relying on any handcrafted feature extraction techniques. The main contributions in this work are as follows:
      
  
      
  	
          
    * We propose a Convolutional Neural Network (CNN) based approach for learning a robust speech filterbank, referred to as DeepVOX, directly from raw speech audio. 
          
    * We propose an adaptive triplet mining technique for efficiently training the DeepVOX filterbank in conjunction with 1D-Triplet-CNN <cit.>, a CNN based speech feature embedding technique, to perform speaker verification.
          
    * We experimentally demonstrate the compatibility and the associated performance benefits of the DeepVOX features with some existing speaker recognition algorithms such as the xVector-PLDA <cit.> and the iVector-PLDA <cit.>.
          
    * We further study the impact of a large variety of audio degradations, multi-lingual speech data, and varying length speech audio on the representation capability of DeepVOX features.
          
    * Finally, we perform a detailed ablation study to identify the type of speech features extracted by DeepVOX and characterize their frequency-response to a various degraded speech audio.
      
      
      
      
  	
  	
  
  § RELATED WORK
   
  	Speech recognition—i.e., recognition and translation of spoken language into machine-readable format—has been one of the most popular tasks in the speech processing community for decades. Therefore, most of the initial speech feature representations were developed from the speech recognition perspective. The widely popular Mel-Frequency Cepstral Coefficients (MFCC) was initially proposed for performing monosyllabic word recognition and was later observed to be efficient for performing speaker recognition as well <cit.>. However, MFCC features are not robust to audio degradations <cit.> and are, therefore, not very suitable for speaker recognition tasks in presence of noisy speech data. This lack of generalizability in the performance of the handcrafted features, such as MFCC, primarily stems from the fact that they are derived from auditory experiments of limited scale. Although these auditory experiments have been later revised multiple times <cit.> to improve their robustness to a wider variety of audio conditions, the scope for improvement remains. This motivated the development of robust speech features for performing speaker recognition in varied non-ideal audio conditions. These speech features are also adept at encoding various physical and acoustic properties of human voice and can be accordingly partitioned into several feature categories, as follows <cit.>: 
  	
  	
  		
    * Short-term spectral features encode the vocal tract shape.
  		
    * Vocal source features characterize the glottal excitation signal.
  		
    * Prosodic features model the speaking style of a speaker.
  		
    * High-Level Features model the lexicon of a speaker.
  	
  	
  	According to the source-filter model of speech <cit.>, human vocal tract can be assumed to behave like a time-varying digital filter due to the articulatory movements. Therefore, in order to model the vocal tract, short-term audio frames (usually 25 to 50ms) are used for extracting the stable voice characteristics in the form of short-term spectral features. Majority of the popular techniques for extracting stable voice characteristics are based on either MFCC or Linear Predictive Coding (LPC) <cit.>. The MFCC feature extraction process uses triangular-filters placed on the Mel-scale for modeling the human auditory perception system <cit.>. The LPC, on the other hand, estimates an all-pole model of filter design for modeling the vocal tract <cit.>.
  	
  	Humans are noted to be efficient in performing speaker recognition in the presence of unknown audio degradations. However, the MFCC feature, which is based on human auditory processing, is unable to cope well in such scenarios <cit.>. Motivated by this, the authors in <cit.> propose the Gammatone Filterbank-based Gammatone Frequency Cepstral Coefficients (GFCC) features as a noise-robust alternative to MFCC. Compared to the Mel-filterbank, the Gammatone Filterbank has finer resolution at lower frequencies, which is claimed to better represent the human auditory model <cit.> and potentially improve speaker recognition performance. Another drawback of the MFCC feature extraction process is its disregard of phase information in the speech data, as the features are extracted only from the amplitude spectrum. The initial motivation behind this was based on human auditory system experiments <cit.>, where short-term phase spectrum did not provide enough performance benefits to justify the associated computational expenses. However, recent studies have reported comparable and complementary speaker recognition performance of both magnitude- and phase-based features <cit.>.
  
  
      
  	
  	
  			
  	LPC-based methods <cit.> in comparison, attempt to characterize the speech production model using an all-pole filter model. Linear Prediction Cepstral Coefficients (LPCC) are the cepstral representation of LPC features and are often considered more reliable than the regular LPC features <cit.>. One of the major disadvantages of the LPC and LPCC-based techniques is that they provide a linear approximation of speech at all frequencies, whereas the spectral resolution of human hearing is known to reduce with frequency beyond 800Hz. This issue was addressed by Hermansky et al. <cit.> in their work on Perceptual Linear Prediction (PLP) Coefficients. For extracting the PLP features on a non-linear scale, that resembles the human auditory system, several spectral transformations <cit.> were applied	to the power spectrum of the speech audio prior to the all-pole model approximation by the autoregressive	model. It is important to note that both LPC and MFCC based features are usually augmented with `spectro-temporal features' in the form of delta and delta-delta coefficients, which are the first and second order time-derivatives of the short-term spectral features, respectively. Spectro-temporal features are one way of adding temporal features, such as formant transitions and energy modulations, to the short-term spectral features.
  
  	The human vocal tract contributes to a majority of the speaker dependent features in the human voice. Short-term spectral features, such as LPC, that attempt to model the human vocal tract are particularly effective in performing speaker recognition. However, vocal tract modeling is not the only way of approaching speaker recognition. Vocal source features <cit.> can also be used for the task. Vocal source features refer to the characteristics of the source of human voice originating in the form of glottal excitation pulses. Features such as glottal pulse shape, rate of vocal fold vibration, and fundamental frequency can potentially be extracted <cit.> to perform speaker recognition. One such work in <cit.> used an LPC-based inverse vocal tract filter and wavelet transform for extracting vocal source features called Wavelet Octave Coefficients Of Residues (WOCOR). It also combined MFCC and WOCOR features for improving overall speaker verification performance.
  	
  
  
  	
  	In the past decade, deep learning based methods have been successfully designed and implemented for solving many speech processing tasks, including speaker recognition <cit.>. A majority of such speaker recognition methods use some type of hand-crafted features, e.g. MFCC, LPCC, as input to their network for solving the problem. For example, authors in <cit.> developed an end-to-end Neural Speaker Embedding System called Deep Speaker that learns speaker-specific embeddings from 64-dimensional log Mel-filterbank coefficients using ResCNN and GRU architectures.
  	However, some of the recent works <cit.> have proposed to feed the raw speech waveform directly as input to deep neural networks for performing a variety of tasks such as speaker recognition and detection of voice presentation attacks. The authors in <cit.>, for example, propose to learn the cut-off frequency of pre-defined band-pass filters for performing speaker recognition on clean (un-degraded) speech data.
  	
  	In this paper, we propose a new approach for extracting robust short-term speech features from raw audio data using 1D-Convolutional Neural Networks (1D-CNN). We draw design cues from our previous work on 1D-CNN <cit.> and 1D-Triplet-CNN <cit.> based architectures for performing speaker identification and verification respectively from degraded audio signals. However, both these architectures use MFCC and LPC-based feature representation as input and are, therefore, limited by the representation power of MFCC and LPC features. We, instead, propose a 1D-CNN based feature extraction module, termed as DeepVOX, to learn and extract speech feature representation directly from raw audio data, in the time-domain itself. The DeepVOX learns filterbanks directly from a large quantity of degraded raw speech audio samples, thereby laying its emphasis on learning robust and highly discriminative speech audio features.
  	
  	Note that, unlike the work in <cit.>, we learn the proposed DeepVOX filterbank without imposing any constraints on the design of the constituent filters. Also, unlike any of the current raw-waveform based speaker recognition methods <cit.>, we demonstrate the compatibility of the proposed DeepVOX features with state-of-the-art deep learning-based speaker recognition methods such as xVectors <cit.> and 1D-Triplet-CNN <cit.> and even on classical methods such as the iVector-PLDA <cit.>. 
  	
  	
  	
  	
  
  § PROPOSED ALGORITHM
  
  	In the previous section, we discussed some popular speech feature extraction techniques. Depending upon the type of the features being extracted, the algorithms were further categorized into four different feature categories. As discussed, human vocal tract significantly contributes to the majority of speaker dependent features in the human voice. Short-term spectral features are, therefore, well-suited for speaker recognition due to their ability to model the human vocal tract. In this work, we propose a method for learning a new type of short-term speech features, referred to as DeepVOX features, using 1D-Convolutional Neural Networks (1D-CNN). It is important to note that, unlike short-term spectral feature extraction algorithms like MFCC, where the extracted speech features are not specifically geared towards speaker recognition, our proposed algorithm learns to extract features directly from raw speech data, specifically suited for the task of speaker recognition. 
  		
      
  
   §.§ Speech Feature Extraction Using DeepVOX
   
      
      In this work, we use the proposed DeepVOX feature extractor jointly with a 1D-Triplet-CNN <cit.>-based feature embedding network for performing speaker recognition. The 1D-Triplet-CNN <cit.> was initially developed for performing speaker verification in degraded audio signals by combining the MFCC and LPC features into a joint-embedding space. However, here the 1D-Triplet-CNN network is used jointly with the DeepVOX to map the DeepVOX features to a highly discriminative speaker embedding space. The proposed joint architecture (see Figure <ref>), also referred to as 1D-Triplet-CNN(DeepVOX), consists of four separate units described below:
      
      
  
    §.§.§ Speech Preprocessing
   
      We first use a Voice Activity Detector <cit.> to remove non-speech parts of an input audio. Any data sample longer than 2 seconds is split into multiple smaller 2 second long audio samples. The resulting speech audio is then framed and windowed into multiple smaller audio clips, called speech units, using a hamming window of length 20ms and stride 10ms, as shown in Figure <ref>. Therefore, each speech unit of duration 20ms sampled at 8000Hz is represented by an audio vector of length 160. The running window extracts a speech unit every 10ms from a 2sec long input audio, thereby extracting around 200 speech units per 2 second long audio sample. These speech units are then stacked horizontally to form a two-dimensional speech audio representation called speech frame, of dimension 160 × 200. The extracted speech frames are then made into speech frame triplets and fed into DeepVOX.
      
      
      
  
    §.§.§ Speech Frame Triplets
  
      The authors in <cit.> introduced the idea of triplet based CNNs. As illustrated in Figure <ref>, our DeepVOX architecture takes a speech frame triplet D_t as input. A speech frame triplet D_t is defined as a tuple of three speech frames: D_t = (S_a,S_p,S_n)
      Here, S_a, the anchor sample, and S_p, the positive sample, are two different speech samples from a subject `X'. S_n, the negative sample, is a speech sample from another subject `Y', such that X ≠ Y.
   
      
  
    §.§.§ DeepVOX
   
      The DeepVOX architecture, as given in Figure <ref>, takes as speech frame triplet as input. DeepVOX processes each speech frame in the triplet to produce a corresponding short term spectral representation, thereby generating a corresponding triplet of DeepVOX features. The design of the DeepVOX architecture primarily comprises of 1D Dilated Convolutional Layers <cit.> and SELU <cit.> (Scaled Exponential Linear Units) non-linearity. The one dimensional filters are so designed that they only learn features from within speech units in a speech frame and not across them. This follows the assumption that the speaker dependent characteristics within each speech unit is independent of other speech units in the speech frame. Each 160 dimensional speech unit within a speech frame is processed by layers of 1D Dilated Convolutional Layers to generate 40 filter responses, which constitute the corresponding short-term spectral representation. These 1D Dilated Convolutional Layers interlaced with SELU non-linearity here are designed to jointly represent a filterbank, which unlike the Mel-filterbank or the Gammatone filterbank, is specifically learned for extracting speaker dependent characteristics.
      
      
  
    §.§.§ 1D-Triplet-CNN
   
      The 1D-Triplet-CNN's architecture comprises of interlaced 1D-Dilated-Convolutional layers and SELU non-linearity, followed by alpha dropout and pooling layers. The use of `dilated convolutions' over `convolutions followed by pooling layers' is motivated by the work done in Wavenet <cit.>, where the authors use dilated convolutions to increase the receptive field size nonlinearly with a linear increase in number of parameters. In context of 1D-Triplet-CNN, 1D dilated convolutions allow the network to learn sparse relationships between the feature values within a speech unit leading to significant performance benefits. The 1D-Triplet-CNN architecture <cit.> is designed for learning speaker dependent speech embedding from triplets of DeepVOX features. The three parallel network branches in the 1D-Triplet-CNN architecture learn and share a common set of weights (see Figure <ref>). The aim of the 1D-Triplet-CNN architecture is to transform the DeepVOX feature triplet input into a triplet of embeddings, where the intra-class samples are embedded closer to each other and inter-class samples are embedded farther apart. This embedding learning process is ensured by the cosine triplet embedding loss.
      
      
      
  
    §.§.§ Cosine Triplet Embedding Loss
   
      The cosine triplet embedding loss <cit.> is a modification upon the triplet loss intially introduced in <cit.> by replacing the euclidean distance metric with cosine similarity. The triplet loss is designed to learn an embedding g(f(x)) ∈^d, where f(x) is DeepVOX feature of speech frame x. In this work, d is set to 128. The embedding is so learned that the intra-class samples are embedded closer to each other than the inter-class samples. The mathematical formulation of cosine triplet embedding loss is given by :
      
      
      L(S_a,S_p,S_n) = ∑_a,p,n^N(cos(g(f(S_a)),g(f(S_n))) - cos(g(f(S_a)),g(f(S_p))) + α_m)
  
      
      Here, L(·,·,·) is the cosine triplet embedding loss function. S_a (the anchor sample) and S_p (the positive sample) are two different speech samples from a subject `X'. S_n (the negative sample) is a speech sample from another subject `Y', such that X ≠ Y. N refers to the total number of triplets in the training set. α_m is the margin of the minimum distance between positive and negative samples and is a user tunable hyper-parameter.
      
      In the training phase, the triplet loss helps the network learn the similarity between the anchor and the positive samples and dissimilarity between the anchor and the negative samples. As illustrated in Figure <ref>, we train both the DeepVOX and the 1D-Triplet-CNN networks together thus simultaneously learning the embedding space using the 1D-Triplet-CNN and the feature space using the DeepVOX.
      
      In the testing phase (see Figure <ref>) we arrange the trained DeepVOX and 1D-Triplet-CNN networks into a siamese network, i.e. only two identical copies of the trained networks are needed. During testing, we input a pair of speech samples into the siamese network to extract a corresponding pair of speech embeddings. The speech embedding pair is then compared using the cosine similarity metric to render a match score. Under ideal conditions, the match score for a genuine pair should be close to 1, while the match score for an impostor pair should be close to -1.
      
      
      
      
      
      
      
      
      
      
  
  
  
  
  
  
  
      
  	
  	
  
    §.§.§ Adaptive Triplet Mining for Online Triplet Selection
   
  	The effectiveness and generalizability of any network trained using the triplet learning paradigm, such as 1D-Triplet-CNN <cit.>, depends on the difficulty of the training triplets. The authors in <cit.> trained their proposed 1D-Triplet-CNN algorithm using offline-generated triplets for performing their speaker recognition experiments. However, the effectiveness and computational-feasibility of offline-triplet generation for evenly sampling a speech dataset drastically reduces with the increase in the number of training samples. Online-triplet generation is, therefore, chosen to effectively train the 1D-Triplet-CNN for our experiments. While the majority of online-triplet generation techniques use either hard or semi-hard triplet mining <cit.>, we propose a curriculum learning-based <cit.> adaptive triplet mining technique. 
  	
  	In adaptive triplet mining, at a given epoch i, the goal is to select a negative sample S_n^i, such that:
  	
      
      .0141!cos(g(f(S_a^i)),g(f(S_p^i))) > cos(g(f(S_a^i)),g(f(S_n^i))) + α_m.
  
      
       
      τ_S_n^i > τ_S_n^i-1
  
      
      Here, S_a^i is the anchor speech sample, S_p^i is the positive speech sample , and α_m is the margin. Here, τ_S_n^i is a parameter that denotes the average difficulty of S_n^i (a negative sample), chosen at epoch i. The difficulty of a negative sample is computed using its cosine similarity to the corresponding anchor speech sample in the triplet. Harder negative samples typically have higher cosine similarity to the corresponding anchor samples, making them harder to separate from the anchor samples. A value of τ = 0 yields the easiest negative sample and τ = 1 yields the hardest negative sample. In our experiments, the value of τ is determined by the current stage (or epoch) of the training process. We initialize the training with the value of τ at 0.4 (empirically chosen) and increase it gradually to 1.0 through the course of the training. This is done to ensure a minimum difficulty of the training triplets at the beginning of the training which is gradually increased as the training proceeds. This helps in avoiding the problem of bad local minima caused by introducing harder negative triplets directly at the beginning of the training <cit.>. It is also observed that learning only on easy and semi-hard triplets lead to poor generalization capability of the model on harder evaluation pairs. Additionally, the model is pre-trained in the identification mode to ensure easier initialization of the training process.
      
      
  	
      
      
  
   §.§ Analysis of the DeepVOX Architecture
  
      
      In Section <ref>, we introduced the DeepVOX architecture for extracting short-term speech features. In this section, we mathematically analyze the proposed architecture and compare DeepVOX's feature learning process with some popular short-term spectral feature extraction algorithms such as MFCC, PNCC, PLP and MHEC. However, before proceeding with the mathematical analysis of DeepVOX's network architecture, we first draw a visual comparison with some popular short-term spectral feature extraction algorithms in Figure <ref>. The main purpose of this comparison is to identify the building blocks of different short-term spectral features and develop an understanding of their individual roles in the feature extraction process. We further use this comparative study to explain the similarities and dissimilarities between our proposed algorithm and some of the existing short-term spectral feature extraction algorithms. 
      
      Furthermore, please note that the DeepVOX method is proposed as an alternative for short-term spectral features such as MFCC and LPC and is intended to be used alongside feature embedding methods such as xVector, iVector, or 1D-Triplet-CNN for performing speaker recognition. Therefore, DeepVOX, similar to MFCC and LPC, is strictly a short-term time-domain feature extraction method, whereas xVector, iVector, and 1D-Triplet-CNN are speech feature embedding methods. Additionally, DeepVOX features, unlike the xVector embeddings, are not a mid-level representation drawn from an end-to-end speaker recognition neural network. Instead, DeepVOX is an independent neural network model carefully designed to learn a time-domain speech filterbank directly from raw audio data. Such an approach makes the DeepVOX features, unlike existing deep learning-based speech embedding networks <cit.>, a direct alternative for short-term spectral features such as MFCC and LPC in speaker recognition models. We specifically trained xVector and iVector models using DeepVOX features to demonstrate its compatibility with existing deep learning-based and classical speaker recognition methods. The experimental results given Section <ref> show its performance benefits over MFCC, LPC, and MFCC-LPC features.
      
      
  
    §.§.§ Building Blocks of Short-term Spectral Feature Extraction Algorithms
  
      
      The comparison in Figure <ref> highlights some key components, given below, important for designing a short-term spectral feature extraction algorithm.
      
      
      
      
      	
    * Pre-emphasis: In the pre-emphasis phase, the speech signal is high-pass filtered to compensate for the natural suppression of high frequency components in the human voicebox. However, this step can degrade the speech quality if the input audio has high-frequency noise and is therefore skipped in our proposed method.
      	
      	
    * Framing and Windowing: Next, the speech signal is split into smaller short-term audio frames, typically 20-30ms long. This is done to reliably extract speaker-dependent vocal characteristics, which are stable only within such short-term frames. In our case, we use a hamming window of length of 20ms and a stride of 10ms for framing and windowing the speech signal.
      	
      	
    * Fourier Transform: FFT (Fast Fourier Transform) is performed to decompose a speech signal based on its frequency content. However, in our case, instead of decomposing the speech frames into their frequency components using FFT, DeepVOX learns speech features in the time domain itself.
      	
      	
    * Filterbank Integration: The FFT response is usually processed through varied handcrafted filterbanks (such as Mel filterbank) for extracting the individual speech features. However, for DeepVOX, instead of using handcrafted filterbanks, a non-linear combination of multiple convolutional filters is used to learn filterbanks specifically suited for performing speaker recognition.
      	
      	
    * Nonlinear Rectification: This step is done to compress the dynamic range of filterbank energies to improve speaker recognition performance <cit.>. However, for the DeepVOX there is no need for an explicit non-linear rectification step due to the inherent non-linearity in the network architecture.    	
      	
      
      
      
      
  
    §.§.§ Mathematical Analysis of the DeepVOX Architecture
  
      
       Majority of the popular short-term spectral feature extraction algorithms such as MFCC and PNCC extract the speaker dependent features from a speech signal using handcrafted filterbanks. To this effect, the Fourier Transform is used to decompose a speech signal into its constituent frequencies, thereby, making filtering operation easier, both semantically and computationally. As the computationally-expensive convolution operation, between the signal and the filter, in time domain is replaced by pointwise multiplication in the frequency domain. The Fourier Transform is usually implemented using the Fast Fourier Transform (FFT) algorithm which makes the filtering of 1D audio signals even more computationally efficient, 𝒪(nlogn), as compared to general convolution operation, 𝒪(n^2). However, FFT only provides a close approximation of time domain filtering and is often inconsistent across different implementations <cit.>, thereby enforcing a trade-off between computational complexity and accuracy. Furthermore, the recent development of efficient GPU-driven implementations of the convolution operation makes Convolutional Neural Networks (CNN) extremely well-suited for performing time domain filtering. Therefore, we use CNN in our algorithm to learn time-domain filters efficiently from raw speech audio. 
  	  
  	  
  	 As discussed earlier and illustrated in Figures <ref> and <ref>, our proposed DeepVOX architecture takes a 2D speech frame S derived from raw speech waveform,  as input to the network. A speech frame S can be represented as:
  	 
      S =  u_1, u_2, ⋯, u_i, ⋯, u_n .
  
  	  
  	 Here u_i is the i^th speech unit and n is the total number of speech units, in the speech frame S. Furthermore, the network outputs a 40 channel filter response f_i corresponding to every speech unit u_i. Therefore, DeepVOX's output O can be given by:
  	  
  	 
      O =  f_1, f_2, ⋯, f_i, ⋯, f_n .
  
  	 
  
  	
  
  	
      f_i    = [ x_i,1, x_i,2, …, x_i,j, …, x_i,40. ]^⊤
  
  	 
  	 Here, x_i,j is the j^th channel filter output for i^th speech unit u_i. In the DeepVOX model, channel outputs at the final layer are results of multiple convolutions of the input data with different convolution filters in the network. Therefore, the network output f_i corresponding to speech unit u_i can be written as:
  	 
  	 
      f_i = (l_m(l_m-1( ⋯ l_k(⋯ l_1(u_i)).
  
  	  
       Here, l_k() is the k^th layer output of the DeepVOX model and m is the total number of layers. Each layer of DeepVOX learns a multi-channel convolutional filter C_k. We can represent l_k() as:
        
       
      l_k(u_i) = C_k ⊛ u_i,
  
        
       Here C_k is the convolutional filter for the k^th layer. The operation in Eq.(<ref>) is equivalent to time-domain filtering of input signal u_i with filter C_k. Hence, we can rewrite Eq.(<ref>) as:
            
        	
       
      f_i = (C_m ⊛ (C_m-1⊛ ( ⋯ C_k ⊛ (⋯ C_1 ⊛ (u_i)).
  
          
          
       Since the convolution operation is associative, we can rewrite Eq.(<ref>) as:
       
      f_i = (C_m ⊛ C_m-1⊛⋯⊛ C_k ⊛⋯ C_1 )_learned DeepVOX filterbank⊛ u_i;
  	    
  	    
  
  
  
  	 The DeepVOX_filterbank, therefore, is designed to learn a 40 channel convolution filter through a combination of multi-channel time-domain filters learned in different layers of the DeepVOX model. Here, each of the 40 channels represents an individual time-domain speech filter in the DeepVOX_filterbank. 
  	 
  	 
  
  	
  
  § DATASETS AND EXPERIMENTS
  
  	In this work, we perform multiple speaker verification experiments on a variety of datasets and protocols. Primarily, we use the VOXCeleb2 <cit.>, Fisher English Training Speech Part 1 <cit.>, and NIST SRE (2008 <cit.>, 2010 <cit.>, and 2018 <cit.>) datasets for training and evaluating the proposed and baseline speaker verification algorithms. We also create degraded versions of the Fisher and NIST SRE 2008 speech datasets by adding diverse noise data from the NOISEX-92 <cit.> dataset under varying levels of (signal-to-noise ratio) SNR (0 to 20 dB) and reverberations. This is done to evaluate the robustness of our proposed method to diverse audio degradations. Additionally, all the speech datasets were sampled at a rate of 8kHz to match the NIST SRE dataset specifications <cit.>. 
  	
  	
  
  
  		
  
  
  	
  	
  
  	
  	
  
      
      
      
      
      
      
  	
  	
      
      
  	
  
   §.§ Datasets
  
  	
  	
  	
  
    §.§.§ VOXCeleb2 Dataset
  
  	The VoxCeleb2 <cit.> dataset contains short interview video clips of 6,112 celebrities recorded in unconstrained scenarios. The entire VOXCeleb2 dataset contains 145,569 video samples from 5,994 celebrities in the training set and 4,911 videos from the remaining 118 speakers in the evaluation set. However, for keeping the triplet-based training process computationally tractable, we only use speech data from one randomly selected video for each subject. In our experiments, each video in the dataset is processed to extract the speech audio, sampled at 8000Hz, from its audio track. Speech samples longer than 5 seconds are split into multiple non-overlapping 5 second long speech samples.
  	
  
  
  	
  	
  
    §.§.§ Fisher English Training Speech Part 1 Dataset
  
  	The Fisher dataset contains pair-wise conversational speech data, collected over telephone channels, from a set of around 12000 speakers. Since the amount of speech data per speaker varies in the dataset, in order to ensure data balance across different speakers, we choose to work with a subset of 6991 speakers, each having at least 250 seconds of speech audio, across 50 samples, after performing voice activity detection. Further, a random subset of 4500 speakers is chosen to train the models and the remaining speakers form the testing set. As mentioned earlier, we have also added the `F-16' and `Babble' noise from the NOISEX-92 <cit.> noise dataset to the Fisher speech dataset. The resultant `degraded-Fisher' speech dataset was maintained at a SNR level of 10dB. We also added reverberations to the speech data generated in a simulated cubical room of side length 4m.
  	
  	
  
    §.§.§ NIST SRE 2008, 2010, and 2018 Datasets
  
  	We also use the NIST SRE 2008 <cit.> dataset in our experiments, given in Table <ref> and Figure <ref>, to evaluate the performance of our proposed algorithm in the presence of multi-lingual data. For our experiments, we choose a subset of speech data from the `phonecall' and `interview' speech types collected under audio conditions labeled as `10-sec', `long' and `short2'. The chosen data subset contains speech from 1336 speakers out of which a randomly chosen subset of 200 speakers is reserved for evaluation purposes. As mentioned earlier, we also add F-16 and Babble noise at a resultant SNR of 0dB to the NIST SRE 2008 dataset to vastly increase the difficulty of the task. We also perform cross-dataset speaker verification performance evaluation using speech data from all the speakers in the evaluation sets of the NIST SRE 2010 <cit.> and NIST SRE 2018 <cit.> datasets.
  	
  
  	
  	
  	
  	
      
      
  
  	
  
   §.§ Experimental Protocols
   
  	In all the experiments, we ensure disjoint set of speakers in the training and testing sets. For evaluating robustness of our models we perform same-noise, cross-noise and  cross-dataset experiments as shown in Tables <ref>, <ref>, and <ref>. The noise characteristics of the training and testing sets used in the different experiments are given alongside in Tables <ref>, <ref>, and <ref>. For example, in Experiment 3 given in Table <ref>, the model was trained on speech data from the training set of Fisher Speech Dataset degraded with Babble noise, and the evaluation was done on speech data from testing set of Fisher Speech Dataset degraded with F16 noise. Note that, no mention of a noise type, such as in Experiment 1 given in Table <ref>, indicates usage of un-altered speech data from the original dataset. Additionally, we have also conducted speaker verification experiments on a subset of multi-lingual speakers from the NIST SRE 2008 dataset, as shown in Table <ref>, for evaluating the effect of speech language on speaker verification performance. Finally, as illustrated in Figure <ref> and discussed in Section <ref>, we have performed Guided Backpropagation <cit.> based ablation study of the features extracted by trained DeepVOX models, to understand the type of audio features considered important for performing speaker recognition by the DeepVOX model.
  	
  	
  
    §.§.§ Baseline Speaker Verification Experiments
  
  	For establishing baseline speaker verification performance on the VOXCeleb2, Fisher, NIST SRE 2008, 2010, and 2018 speech datasets, we choose iVector-PLDA <cit.> and xVector-PLDA <cit.> algorithms trained on the baseline features (MFCC, LPC, MFCC-LPC) and DeepVOX features separately. This is done to evaluate and compare the effectiveness of DeepVOX features, with respect to baseline features, in both classical and deep learning-based speaker recognition algorithms. However, unlike the baseline features, DeepVOX feature extraction process requires a DeepVOX model to be trained. For each of the experiments in Tables <ref>,  <ref>, and <ref> we use speech data only from corresponding training set to train the DeepVOX model, ensuring disjoint data and subjects in the training and testing sets for the DeepVOX feature extraction process. We also use the RawNet2 <cit.> algorithm for establishing baseline raw audio-based speaker recognition performance.
  	
  
  
  		
  		
    * iVector-PLDA <cit.>-based Speaker Verification Experiments:
  		We use MSR Identity Toolkit's <cit.> iVector-PLDA implementation as our first baseline speaker verification method. A Gaussian-PLDA (gPLDA)-based matcher <cit.> is used to compare the extracted i-Vector embeddings of a pair of speech samples.
  		
  		
  	
  
  
  		
  		
  		
    * xVector-PLDA <cit.>-based Speaker Verification Experiments:
          We use the PyTorch-based implementation <cit.> of the xVector algorithm as our second baseline speaker verification method. A gPLDA-based matcher <cit.> is used to compare the extracted xVector embeddings of a pair of speech samples.
  		
  		
    * RawNet2 <cit.>-based Speaker Verification Experiments:
  		We use the RawNet2 algorithm to establish a baseline raw audio-based speaker recognition performance. We use the authors' <cit.> original implementation of the RawNet2 method for performing the RawNet2-based experiments.
          
  	
  	
  	
  
    §.§.§ Speaker Verification Experiments on 1D-Triplet-CNN Algorithm Using MFCC-LPC Feature Fusion
  
  	We also perform speaker recognition experiments using the 1D-Triplet-CNN <cit.> algorithm. These experiments provide benchmark results (given in Tables <ref>,<ref>, and <ref>) to directly compare the performance of the DeepVOX feature to MFCC, LPC, and MFCC-LPC features in a deep learning framework.
  	For training the 1D-Triplet-CNN, speech audio triplets are formed using the speakers from the training set. The speech audio triplets are then processed to extract 40 dimensional MFCC and LPC features separately. The extracted MFCC and LPC features are then stacked together to form a two-channel input feature patch for the 1D-Triplet-CNN. For evaluation, speech audio pairs are fed to the trained model to generate pairs of speech embeddings. The speech embeddings are then matched using the cosine similarity metric.
  	
  	
  
    §.§.§ Speaker Verification Experiments on 1D-Triplet-CNN Algorithm Using DeepVOX Features (Proposed Algorithm)
  
  	In these set of experiments, we evaluate the performance of our proposed approach on multiple training and testing splits (given in the Tables <ref>,<ref>, and <ref>) drawn from different datasets and noise types and compare it with the baseline algorithms. Similar to the MFCC-LPC-based 1D-Triplet-CNN <cit.> algorithm, our algorithm also trains on speech audio triplets. However, instead of extracting hand-crafted features like MFCC or LPC, our algorithm trains the DeepVOX and 1D-Triplet-CNN modules together to learn both the DeepVOX-based feature representation and 1D-Triplet-CNN-based speech feature embedding simultaneously. For evaluation, speech audio pairs are fed to the trained DeepVOX model to extract pairs of DeepVOX features which are then fed into the trained 1D-Triplet-CNN model to extract pairs of speech embeddings and compare them using the cosine similarity metric.
  	
  	
  
    §.§.§ 1D-Triplet-CNN-based Speaker Recognition Experiments Using Adaptive Triplet Mining
  
  	The proposed adaptive triplet mining technique is evaluated by repeating all the 1D-Triplet-CNN based speaker verification experiments on MFCC, LPC, MFCC-LPC, and DeepVOX features, referred to as 1D-Triplet-CNN-online in Tables <ref>, <ref>, and <ref>. In our experiments, the 1D-Triplet-CNN models are pretrained in identification mode for 50 epochs followed by 800 epochs of training in verification mode using adaptive triplet mining. As also mentioned in Section <ref>, the difficulty (τ) of the mined negative samples is gradually increased from 0.4 to 1.0 linearly over 800 epochs. Also, it is important to note that the triplet mining is done in mini-batches of 6 randomly chosen samples drawn from each of the 25 randomly chosen training subjects. 
  	
  	
  	
  
    §.§.§ Effect of Language on Speaker Verification
  
  	The effect of language on speaker recognition performance, also known as the language-familiarity effect (LFE), of both humans and machines, has been studied in the literature <cit.>. According to LFE, human listeners perform speaker recognition better when they understand the language being spoken. Similar trends have been noticed in the performance of automatic speaker recognition systems <cit.>. In this work, we perform additional speaker recognition experiments (Exp. #  12 to 14 in Table <ref>) on a subset of the NIST SRE 2008 dataset for evaluating the robustness of the DeepVOX features compared to MFCC, LPC, and MFCC-LPC features in the presence of multi-lingual speech data. In all the experiments (Exp. # 12 to 14), the models are trained on English speech data spoken by a subset of 1076 English-speaking subjects in NIST SRE 2008's training set and evaluated on a subset of 59 multi-lingual subjects, containing speech data from 15 different languages, in NIST SRE 2008's test set. The evaluation trials in experiments 12 to 14 varied as follows:
  	
      
          
    Same language, english only trials :  In Exp. # 12, the trained models are evaluated on same-language (English Only) trials. This experiment establishes the baseline same-language (English to English) speaker verification performance of all the algorithms.
          
    Same language, non-english trials:  In Exp. # 13, the trained models are evaluated on same-language (Multi-lingual) trials. This experiment aims to investigate the performance of speaker recognition models trained on English-only speech data for matching Non-English same-language (e.g: Hindi to Hindi) speech trials.
          
    Cross-lingual trials:  In Exp. # 14, the trained models are evaluated on different-language (Cross-lingual) trials. This experiment aims to investigate the performance of speaker recognition models trained on English-only speech data for matching Non-English different-language (e.g., Chinese to Russian) speech trials.
      
  	
  
  	
  
    §.§.§ Effect of Audio Length on Speaker Verification
   
  	The reliability of the speaker-dependent features extracted from an audio sample depends on the amount of usable speech data present within, which is directly dependent on the length of the audio sample. Therefore, performing speaker recognition in audio samples of a small duration is a challenging task. Since in real-life scenarios, probe audios are of relatively small audio durations (1 sec - 3 secs), the feature extraction algorithm needs to be able to reliably extract speaker-dependent features from speech audio of limited duration. In this experiment (see Table <ref> and Figure <ref>), we compare the speaker verification performance of our proposed algorithm with the baseline algorithms on speech data of varying duration from the NIST SRE 2008 dataset. The duration of probe audio is varied between 3.5 secs and 0.5 secs in steps of 0.5 secs. 
  	
  	
      
  	
      
      
  
  	
  
  § RESULTS AND ANALYSIS
   
  	
  	The results for all the experiments described in Section <ref> are given in Tables <ref>, <ref>, <ref>, <ref>, <ref>. The content of all the tables is summarised in the DET curves given in Figures <ref>, <ref> to present the results in an easier-to-consume format. For all the speaker verification experiments, we report the	True Match Rate at False Match Rate of 1% and 10% (TMR@FMR={1%,10%}), minimum Detection Cost Function (minDCF) at C_miss (cost of a missed detection) value of 1 and Equal Error Rate (EER, in %) as our performance metrics for comparison of the baseline methods and the proposed method. The minDCF is reported at two different a priori probability of the specified target speaker, viz., P_tar of 0.01 and 0.001 (minDCF(P_tar={0.01,0.001}). The Detection Error Tradeoff (DET) curves are given in Figure <ref>.
  	
  
  	
  	
  	
  		
    * Overall, in all the speaker verification experiments given in Tables <ref>, <ref>, <ref>, <ref>, and <ref>, the 1D-Triplet-CNN algorithm using DeepVOX features trained with adaptive triplet mining, also referred to as 1D-Triplet-CNN-online(DeepVOX), performs the best. The proposed adaptive triplet mining
  		method improves the verification performance (TMR@FMR=1%) of the 1D-Triplet-CNN algorithm using DeepVOX features by 3.01%, and MFCC-LPC features by 8.71%. Similar performance improvements are also noticed for the MFCC and LPC features across all the performance metrics. This establishes the benefits of using the adaptive triplet mining technique over offline-triplet mining for efficiently training the 1D-Triplet-CNN based speaker recognition models.
  		
  		
    * Across all the speaker verification experiments given in Tables <ref>, <ref>, <ref>, <ref>, and <ref>, the second-best performance, after DeepVOX features, is obtained by the feature level combination of MFCC and LPC features, referred to as MFCC-LPC features. Therefore, we choose MFCC-LPC features as our strongest baseline feature. In the upcoming discussions, all performance improvements offered by the DeepVOX features, for any particular algorithm, is reported in comparison to the MFCC-LPC features. Furthermore, we will also draw comparison with the RawNet2 model to establish DeepVOX's performance benefits over a current state-of-the art raw speech audio-based speaker recognition method.
  		
  	    
    * In experiment #1 on the VOXCeleb2 dataset, given in Table <ref> and Figure <ref>, the 1D-Triplet-CNN-online(DeepVOX) method performs the best across all the performance metrics. The DeepVOX features improve the verification performance (TMR@FMR={1%,10%}), specifically for the 1D-Triplet-CNN-online algorithm, over the best performing baseline feature (MFCC-LPC) by {9.89%,0.9%}. It also reduces the EER by 2.5% and minDCF(P_tar={0.001,0.01}) by {0.03, 0.15}. Similarly, for the 1D-Triplet-CNN, xVector-PLDA, and iVector-PLDA algorithm, the DeepVOX features improve verification performance over the best performing baseline feature (MFCC-LPC). The 1D-Triplet-CNN(DeepVOX) method also outperforms the RawNet2 across all the performance metrics. The TMR@FMR={1%,10%} is increased by {0.23%,0.97%}, EER is reduced by 0.99%, and minDCF(P_tar={0.001,0.01}) is reduced by {0.026,0.02}.
  	    
  	    
  	   
  	    
      
      	
      	
    * In all the four speaker verification experiments (Experiments 2 to 5) on the degraded Fisher dataset given in Table <ref> and Figure <ref>, the 1D-Triplet-CNN-online (DeepVOX) method performs the best across all the performance metrics. It is important to note that the performance of all the algorithms is significantly lower in case of cross-noise experiments (Experiments 3 and 5) when compared to the same-noise experiments (Experiments 2 and 4). However, the usage of the proposed DeepVOX features in all the algorithms improves their robustness to the mis-match in the training and testing noise characteristics. Also, the speaker recognition performance in the presence of babble noise, compared to the F-16 noise, is observed to be significantly lower. This indicates speech babble as one of the more disruptive speech degradations for speaker recognition tasks. All the algorithms when trained on DeepVOX features, as compared to MFCC, LPC or MFCC-LPC features, gain significant performance improvements.
      	
      	
    * On an average across the four speaker verification experiments (Experiments 2 to 5) on the degraded Fisher dataset, the incorporation of DeepVOX features in the 1D-Triplet-CNN-online algorithm improves the verification performance (TMR@FMR={1%,10%}) over the MFCC-LPC feature by {23.83%,10.65%}, reduces the EER by 5.98%, and improves minDCF(P_tar={0.001,0.01}) by {0.007,0.24}. Similarly, for the 1D-Triplet-CNN, xVector-PLDA, and iVector-PLDA algorithm, the DeepVOX features improve speaker verification performance over the best performing baseline feature (MFCC-LPC). The 1D-Triplet-CNN(DeepVOX) method also outperforms the RawNet2 across all the performance metrics. The TMR@FMR={1%,10%} is increased by {25.32%,17.62%}, EER is reduced by 10.21%, and minDCF(P_tar={0.001,0.01}) is reduced by {0.004,0.14}. Furthermore, the proposed method's performance benefits compared to the RawNet2 is even greater in the cross-noise experiments (Experiments 3 and 5), demonstrating its superior resilience to mis-matched degraded audio conditions.
      	
      	
      
      	
      
      	
      	
    * On an average across the seven speaker verification experiments (Experiments 6 to 12), all the algorithms gain performance benefits when the MFCC, LPC and MFCC-LPC features are replaced with DeepVOX features for training the models. Replacing the best performing baseline feature (MFCC-LPC) by DeepVOX features in the 1D-Triplet-CNN-online algorithm improves the verification performance (TMR@FMR={1%,10%}) by {14.72%,8.7%}, reduces the EER by 3.67% and minDCF(P_tar={0.001,0.01}) by {0.009,0.11}. Similarly, for the 1D-Triplet-CNN, xVector-PLDA, and iVector-PLDA algorithm, the DeepVOX features improve speaker verification performance over the best performing baseline feature (MFCC-LPC). The 1D-Triplet-CNN(DeepVOX) method also outperforms the RawNet2 across majority of the performance metrics. The TMR@FMR={1%,10%} is increased by {5.57%,10.6%}, EER is reduced by 3.29%. However, no significant change in minDCF(P_tar={0.001,0.01}) was observed. The 1D-Triplet-CNN(DeepVOX) method also vastly outperforms the RawNet2 method in cross-noise experiments (Experiments 11 and 12) on the degraded NIST SRE 2008 dataset.
      	
      	
      
      	
      	
  
  		
  		
    * In the three speaker verification experiments (Experiments 13 to 15, given in Table <ref>) on multi-lingual speakers from the NIST SRE 2008 dataset, DeepVOX features perform the best across all the algorithms and metrics. The incorporation of DeepVOX features, compared to the MFCC-LPC features, in the 1D-Triplet-CNN-online algorithm, improves the verification performance (TMR@FMR={1%,10%}) by {23.95%,8.97%}, reduces the EER by 4.99% and minDCF(P_tar={0.001,0.01}) by {0.02,0.21}. Similar performance benefits of DeepVOX features were noted for the 1D-Triplet-CNN, xVector-PLDA, and iVector-PLDA algorithms, as well. The 1D-Triplet-CNN(DeepVOX) method also outperforms the RawNet2 across all the performance metrics. The TMR@FMR={1%,10%} is increased by {10.18%,5.19%}, EER is reduced by 3.24%, and minDCF(P_tar={0.001,0.01}) is reduced by {0.019,0.12}.
  		
  		
  
  		
  		
    * It is interesting to note the effect of language on verification performance in Experiments 13 to 15. Best speaker verification performance is achieved in Experiment 13, where the models are trained on English speech data and evaluated on same-language English-only speech audio pairs. However, introduction of same-language multi-lingual speech audio pairs to the evaluation set (in Experiment 14) reduces the verification performance (TMR@FMR=1%) of 1D-Triplet-CNN-online by 3.70% for the DeepVOX features, 14.28% for the MFCC-LPC features, 4.11% for the MFCC features, and 17.46% for the LPC features. Furthermore, re-evaluating the same models on cross-language multi-lingual speech audio pairs in Experiment 15 results in the largest reduction in verification performance, verifying the impact of language-familiarity effect <cit.> in all algorithms and features used in our experiments. It is important to note that the detrimental effects of the language-familiarity effect (in Experiment 14) are observed to be the weakest at 22.49% (performance reduction (TMR@FMR=1%)) for the DeepVOX features compared to 40.76% for the MFCC-LPC features, 39.31% for the MFCC features, and 37.91% for the LPC features, using the best-performing 1D-Triplet-CNN-online algorithm.
  		
  		
    * In the experimental results given in Table <ref> and illustrated in Figure <ref>, we notice a gradual decrease in verification performance (across all algorithms and features) with the decrease in length of audio samples in the testing data. However, the loss in performance is observed to be much lower with the usage of DeepVOX features compared to MFCC, LPC, or MFCC-LPC features across all the algorithms. The 1D-Triplet-CNN-online algorithm using DeepVOX features sufferes a performance (TMR@FMR=10%) reduction of 10%, compared to a reduction of 32% using MFCC-LPC features, 45% using MFCC features, 34% using LPC features, when the audio length is reduced from 3.5 seconds to 0.5 seconds. Similar trends were observed for the 1D-Triplet-CNN, xVector-PLDA, and the iVector-PLDA algorithms across the DeepVOX, MFCC-LPC, MFCC, and LPC features. For the RawNet2 algorithm, a performance loss of 48% is observed when the length of raw input audio is reduced from 3.5 seconds to 0.5 seconds. It is important to note that when compared to the 1D-Triplet-CNN based algorithms, relatively larger performance losses are observed for the iVector-PLDA, xVector-PLDA, and RawNet2 algorithms, across all the features. However, using the DeepVOX features improves the robustness of even the iVector-PLDA and xVector-PLDA algorithms when performing speaker verification on speech samples of limited duration, thereby asserting the effectiveness of the DeepVOX features in the task.
  		
  		
  		
  
  		
  
  		
  	
  	
  
  	
  	
  
  
  
  
  
  
  
  
  
  
  
  
  
  	
  	
      
      
      
      
  	
      
  	
  	
  
  § ABLATION STUDY OF DEEPVOX
   
  	In this section, we  use `Guided Backpropagation' <cit.> to analyze the type of speech information being extracted by the DeepVOX feature. Such an analysis reveals the components of a speech audio that are deemed important, by the DeepVOX model, in the context of speaker recognition. In this analysis, we use the DeepVOX model trained for Experiment #1 on the VOXCeleb2 dataset, due to diverse speakers and recording conditions in the training data. For evaluation, we choose audio samples from the TIMIT <cit.> dataset due to the availability of ground-truth information for analysis of frequency sub-bands essential for speaker recognition <cit.>. For analysing the DeepVOX method, we feed an input audio sample to the trained DeepVOX model and extract the 40-dimensional DeepVOX features. Guided backpropagation is then used individually on each of the 40 features to estimate the corresponding relevance signals. The relevance signal in this case refers to the portion of input audio signal (in the frequency domain) that the DeepVOX model fixates on to extract a corresponding DeepVOX feature. The 40 relevance signals corresponding to the 40 DeepVOX features are aggregated to estimate the mean relevance signal. The mean relevance signal is then analysed, as given below, to characterize the properties of the speech signal extracted by the DeepVOX features important for performing speaker recognition:
  	
  	
      
    Fundamental Frequency (F0) Extraction by the DeepVOX: In this experiment, we extract speech utterances corresponding to the five phonemes /ah/, /eh/, /iy/, /ow/, /uw/ from a randomly chosen speaker in the TIMIT dataset. The speech audio of these phonemes is then fed to the trained DeepVOX model to extract corresponding DeepVOX features and subsequently extract the corresponding relevance signals. The input speech signal and the corresponding mean relevance signal are then compared using the Praat <cit.> toolkit (see Figure <ref>). While the waveform representation of the original input signal and the corresponding mean relevance signal differ visually, pitch contour analysis of the signals reveals that the relevance signal successfully captures the F0 information from the input speech signal. This indicates that the DeepVOX architecture successfully extracts and uses fundamental frequency (F0) (a vocal source feature), for representing the human voice. This could be seen as a direct effect of the presence of phase information in the raw input speech audio, as phase information in speech audio captures rich vocal source information <cit.>.
      
      
    Operational Frequency-range of DeepVOX: Similar to <cit.>, we plot the input audio signal (in red color) and corresponding relevance signal (in blue color) on the Power Spectral Density (PSD) plots (given in Figure <ref>). The PSD plots are inspected for frequency-band overlap in the input audio signal and the corresponding mean relevance signal. The overlap indicates the frequency components of the input audio signal that DeepVOX captures for performing speaker recognition. As observed in Figure <ref>[a], the trained DeepVOX model reliably models a clean speech input signal in the frequency range of 0 to 4000Hz, with better modeling performance observed in the range of 2000Hz to 4000Hz, which is known to contain highly discriminative speaker-dependent information <cit.>. This demonstrates DeepVOX's ability to use spectral information in the frequency range of 0 to 4000Hz for performing speaker recognition.
  
      
    Effect of Audio Degradation on the DeepVOX:  Furthermore, as shown in Figure <ref> [(b)], the trained DeepVOX model also reliably models a speech signal degraded with synthetic car noise from the NOISEX-92 dataset <cit.>. However, it fails to model the synthetic car noise in absence of speech, as shown in Figure <ref> [(c)]. This demonstrates DeepVOX's ability to selectively model the speech components and reject the background noise in an audio sample for performing speaker recognition.
      
      
      
      
      
    Layer-wise magnitude frequency response of the DeepVOX: Finally, we also plotted (see Figure <ref>) the layer-wise cumulative magnitude frequency response of the convolution filters in the DeepVOX model trained on the VoxCeleb2 dataset. Here we observed that while the initial three layers behave as a multi-band pass filter, the later layers act as low-pass filters. Specifically, the first three layers' cumulative magnitude frequency response shows peaks in the frequency range of 0-800Hz and 1500-3000HZ. Comparing to the acoustic characteristics of the human voice in American English <cit.>, the first peak (0-800Hz) is specifically suited for capturing the fundamental frequency (F0) and first formant (F1) of the human voice (the average F0 is 195Hz and average F1 is 595Hz) and the second peak (1500-3000HZ) can capture the second (F2) and third (F3) formants of the human voice (the average F2 is 1734Hz and the average F3 is 2826Hz). Therefore, the initial layers of the DeepVOX model learn to capture important speaker-dependent speech characteristics (F0, F1, F2, and F3) from input speech audio and are well-suited for application in a speaker recognition system.
      
      
      
  
  
  
      
  
  	
  
  § APPLICATIONS OF DEEPVOX
  
  	
  
  
  	Speaker recognition systems find applications in several different domains, including telephone banking <cit.>, E-Commerce <cit.> and forensics <cit.>, and personal virtual assistants <cit.> in the form of voice-controlled user interfaces. However, with the increase in the applications of speaker verification technology, its surface area for incoming threats of circumvention or misuse is also increased. For example, the authors in <cit.> developed an adversarial audio sample that can be played to stop Amazon Alexa from being activated, thus launching a form of denial-of-service attack (DoS attack). Another form of attack called voice spoofing <cit.> can be used to impersonate a target user and fraudulently gain access to sensitive user data. Recently, with the advent of DeepFake technology, it is now possible to synthesize realistic speech audio or create convincing alterations of existing speech audios in the public domain to cause widespread panic and confusion <cit.>. In such scenarios, a robust speaker recognition system, such as DeepVOX, can help thwart the attempts of unauthorized users to illegitimately access a voice interface through voice spoofing or launching DOS attacks. 
  
      Towards that end, in this work, we specifically evaluated DeepVOX's generalizability across a wide variety of speech audio, ranging from telephonic speech conversations in the Fisher Speech Corpora to the nearly-unconstrained interview speeches from the VOXCeleb dataset. Furthermore, we also explored a wide variety of speech audio degradations and assessed their impact on the speaker verification performance of DeepVOX-based models. We specifically introduced the experiments with a mismatch in the audio degradations in the train and evaluation sets (Experiments 3 and 5 in Table <ref>) and mismatch in spoken language in experiments 14 and 15 in Table <ref> to simulate the effect of domain mismatch on the speaker verification performance. While we do notice a performance drop in case of domain mismatch across all the methods and feature combinations tested in this work, the negative impact of domain mismatch is notably reduced across all the scenarios when the DeepVOX replaces traditional speech features (MFCC and LPC). This demonstrates that the DeepVOX in its current form is relatively robust to the adverse effects of mismatch between the training and testing conditions and can be applied to a wide variety of applications discussed above, where domain mismatch is expected. Furthermore, we believe it is possible to tweak the DeepVOX hyperparameters such as the number of DeepVOX filterbanks, the type, length, and stride of the windowing function to adapt the DeepVOX method to specific datasets and audio conditions and further improve its performance.
      
      Additionally, from an implementation perspective, it is important to note that we designed the DeepVOX to offer either an end-to-end learnable or a drop-in replacement for handcrafted filterbanks such as MFCC. For example, on the one hand, we can train DeepVOX end-to-end with any speaker embedding extraction system, as our experiments do with the 1D-Triplet-CNN system. On the other hand, we can replace a fixed MFCC-based feature extraction pipeline with a pre-trained DeepVOX filterbank to asynchronously train a speech embedding method, as shown in our experiments with the xVector and iVector based methods. Also, note that DeepVOX features are generated at the frame level like traditional MFCC features and it should not be confused with a fixed-dimensional speech embedding extractor such as xVector. Therefore, in summary, DeepVOX provides a learnable a time-domain speech filter bank that can either be used to train robust end-to-end speaker recognition systems from scratch or retrofit into existing speaker recognition frameworks. Furthermore, the DeepVOX-based speaker recognition system's robustness to various non-ideal audio conditions, such as background noise, language mismatch, and short audio duration, makes it an essential tool in the arsenal of digital audio forensics to protect and verify the integrity of data before using it.
  
      
  	
  	
  
  	
  
  § CONCLUSION
  
  	
  
  	The performance of short-term speech feature extraction techniques, such as MFCC, is dependent on the design of handcrafted filterbanks such as the Mel filterbank. While such techniques are easy to use and do not require any training data, they do not adapt well to diverse non-ideal audio conditions. Therefore, it is beneficial to develop feature extraction techniques, such as DeepVOX, that can robust across diverse non-ideal audio conditions, as evident in the experimental results. The frequency analysis of the learned DeepVOX filterbanks indicates that it can extract spectral information from a large frequency range (0 to 4000Hz) and also extract the fundamental frequency (F0) information for representing the speaker in speech audio. It is also important to make note of rare cases such as Experiment 8 in Table <ref>, where certain combinations of noise characteristics in the training and testing sets create challenging scenarios for the proposed DeepVOX. Therefore, it is important to continue research in the development of feature extraction algorithms that builds upon and improves the currently proposed algorithm. Furthermore, as discussed in Section <ref>, the DeepVOX algorithm has a limitation of being trained only 200 audio frames at a time; hence, it cannot benefit from training on longer audio samples. We plan to extend our DeepVOX model by incorporating methods for automatically learning from audio samples of varying lengths.
  	
  	
  
  	
  
  § ACKNOWLEDGMENT
  
  	We thank Dr. Shantanu Chakrabartty and Dr. Kenji Aono from Washington University in St. Louis for providing us their audio degradation tool, used in this work. We also thank Dr. Joseph P. Campbell from MIT Lincoln Lab for the useful discussions.
  	
  	
  	
  	
  	
  	
  
  	ieee
  		
  	
  	
  
  
  
  
  
  
      
  
  	
  	
  