Title of Invention: Method and system for reducing interference and noise in speech signals
Abstract: Interference in an audio signal is reduced by estimating a target signal using beam-forming in a direction of the signal source. A set of estimates of interference is determined by using a microphone array filtering matrix to block the target signal in the audio signal. A set of filters is optimized by minimizing an objective function measuring a mismatch between the set of estimates of interference and the estimate of the target signal. The minimizing uses a sparse regularization of coefficients of the set of filters. The set of estimates of interference are filtered using the set of filters after the optimizing. Then, the estimate of interference after the optimizing is subtracted from the target signal.
Detailed Description: Part general_description_paragraphs: Part FIELD OF THE INVENTION: This invention relates generally to processing audio signals, and more particularly to cancelling interference in speech signals acquired by an antenna array.
Part BACKGROUND OF THE INVENTION: Speech-based command interfaces can be used in vehicles. Applications include automatic dialog system for hands-free phone calls, as well as more advanced features, such, as navigation systems.
However, interference, such as speech from the codriver, rear-seat passengers, and noise, e.g., music or radio, engine and wind noise, can significantly degrade performance of an automatic speech recognition (ASR) system, which is crucial for those applications. This issue can be addressed with adaptive interference cancellation techniques, such as generalized sidelobe cancellation (GSC).
Beamformers based on GSC are well known. Typically, the beamformer includes a presteering front end, a fixed beam former (FBF), a blocking matrix (BM) and an adaptive canceller. The presteering front end is composed of various time delays allowing the main lobe of the beamformer to be steered to a selected direction. The FBF is used to enhance a target signal from the selected direction. However, the BM, composed of adaptive blocking filters (ABF), rejects the target signal, so that the blocking matrix contains interference and noise. The adaptive canceller, composed of adaptive canceling filters (ACF), is able to adjust weights so that the interferences and noise can be subtracted from the fixed beamformer output.
However, the conventional adaptive beamformer for GSC, like the simple Griffiths-Jim beamformer (GJBF), see U.S. Patent Applications 20100004929, 20070244698, 20060195324 and 20050049864 D. suffers from target signal cancellation due to steering-vector errors. The steering-vector errors are due to errors in microphone positions, microphone gains and real world recordings of, e.g., reverberation, noise and a moving target. Indeed, the beamformer is constrained to produce a dominant response toward the selected location of the source of the speech, while minimizing the response in all other directions.
However, in reverberant environments a single direction of arrival cannot be determined because the desired signal and its reflection impinge on the microphone array from several directions. Thus, complete rejection of the target signal is almost impossible in the BM and a considerable portion of the desired speech is subject to interference cancellation, which results in target signal cancellation.
In addition, the original formulation of the GJBF was based on the general use of beamforming, where the far-field assumption is often valid, such as in radio astronomy or geology.
However, in a vehicle, the microphone array can span about one meter, meaning that the far field assumption is no longer valid. This change in the physics of the system also causes leakage in the conventional Griffiths-Jim BM because now the target signal is no longer received at each microphone with equal amplitude.
Applying the GSC uniformly to an entire utterance, without considering the observed data, is not efficient. It is not necessary to process noise only and single speaker segments using the GSC if they can be accurately labeled.
In particular, non-overlapping speech, and non-speech occur more often than overlapping speech, and each case needs to be handled differently
GSC
FIG. 1 shows a conventional GSC, which is a simplification of the well known Frost Algorithm. It is assumed that all input channels 101 have already been appropriately steered toward a point of interest. The GSC includes an upper branch 110, often called the Fixed Beamformer (FBF), and a lower branch 120 including a Blocking Matrix (BM) outputting to normalized least mean square modules 140, whose outputs are also summed 150.
The conventional Delay and Sum beamformer for FBF is to sum 130 the observed signals xm from the microphone array as




  
    
      
        
          
            
              y
              FBF
            
            ⁡
            
              (
              t
              )
            
          
          =
          
            
              1
              M
            
            ⁢
            
              
                ∑
                
                  m
                  =
                  1
                
                M
              
              ⁢
              
                
                  x
                  m
                
                ⁡
                
                  (
                  
                    t
                    -
                    
                      τ
                      m
                    
                  
                  )
                
              
            
          
        
        ,
      
    
    
      
        (
        1
        )
      
    
  




where τm is the delay for the mth microphone, for a given steering direction.

The lower branch utilizes an unconstrained adaptive process on a set of tracks that have passed through the BM, including of some process intended to eliminate the target signal from the incoming data in order to form a reference of the noise. The particular Griffiths-Jim BM takes pairwise differences of signals, which can be expressed for a four-microphone instance as




  
    
      
        
          W
          b
        
        =
        
          (
          
            
              
                1
              
              
                
                  -
                  1
                
              
              
                0
              
              
                0
              
            
            
              
                0
              
              
                1
              
              
                
                  -
                  1
                
              
              
                0
              
            
            
              
                0
              
              
                0
              
              
                1
              
              
                
                  -
                  1
                
              
            
          
          )
        
      
    
    
      
        (
        2
        )
      
    
  




For this Wb the BM output tracks are determined as the matrix product of the BM and matrix of current input data

yBM(t)=WbX(t)  (3)

where X(t)=[x1(t), x2(t), . . . , xM(t)]. The overall beamformer output y(t) 102, is determined as the DSB signal minus 160 the sum 150 of the adaptively-filtered BM tracks





  
    
      
        
          y
          ⁡
          
            (
            t
            )
          
        
        =
        
          
            
              y
              FBF
            
            ⁡
            
              (
              t
              )
            
          
          -
          
            
              ∑
              
                m
                =
                1
              
              
                M
                -
                1
              
            
            ⁢
            
              
                ∑
                
                  i
                  =
                  
                    -
                    
                      K
                      L
                    
                  
                
                
                  K
                  R
                
              
              ⁢
              
                
                  
                    g
                    
                      m
                      ,
                      i
                    
                  
                  ⁡
                  
                    (
                    t
                    )
                  
                
                ⁢
                
                  
                    
                      y
                      m
                      BM
                    
                    ⁡
                    
                      (
                      
                        t
                        -
                        i
                      
                      )
                    
                  
                  .
                
              
            
          
        
      
    
    
      
        (
        4
        )
      
    
  




Define, for m=1, . . . , M−1

xm(t)=(xm(t+KL), . . . ,xm(t), . . . ,xm(t−KR)),

and

gm(t)=(gm,−KL(t), . . . ,gm,0(t), . . . ,gm/KR(t)),

then the adaptive normalized multichannel least mean square (LMS) solution is





  
    
      
        
          
            
              
                g
                m
              
              ⁡
              
                (
                
                  t
                  +
                  1
                
                )
              
            
            =
            
              
                
                  g
                  m
                
                ⁡
                
                  (
                  t
                  )
                
              
              +
              
                
                  μ
                  
                    
                      p
                      est
                    
                    ⁡
                    
                      (
                      t
                      )
                    
                  
                
                ⁢
                
                  
                    x
                    m
                  
                  ⁡
                  
                    (
                    t
                    )
                  
                
                ⁢
                
                  y
                  ⁡
                  
                    (
                    t
                    )
                  
                
              
            
          
          ;
        
        ⁢
        
          

        
        ⁢
        
          
            m
            =
            1
          
          ,
          …
          ⁢
          
            
          
          ,
          
            M
            -
            1
          
          ,
          
            

          
          ⁢
          where
        
      
    
    
      
        (
        5
        )
      
    
  
  
    
      
        
          
            p
            est
          
          ⁡
          
            (
            t
            )
          
        
        =
        
          
            ∑
            
              m
              =
              1
            
            M
          
          ⁢
          
            
              
                
                
                  
                    x
                    m
                  
                  ⁡
                  
                    (
                    t
                    )
                  
                
                
              
              2
              2
            
            .
          
        
      
    
    
      
        (
        6
        )
      
    
  




Part SUMMARY OF THE INVENTION: The embodiments of the invention provide a system and method for reducing interference and noise in audio signals including speech using generalized sidelobe cancellation (GSC). In the preferred embodiment, the GSC is embedded in a multi-stage system designed to distinguish noise only segments (NO), single speaker segments (SS), and overlapping speech segments (OS), and then apply different approaches to suppress diffuse noises and other interferences within different types of the segments. In the overlapping speech segments, speech other than from a target speaker is considered interference.
A first stage is carried out by overlapping speech detection (OSD) and a voice activity detection, while the second stage is achieved by a blind speech separation (BSS) based speaker turn detection, direction of arrival estimation, sparse GSC and post-processing, i.e., log-spectral amplitude estimation.
Part DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS: The embodiments of the invention provide a system and method for reducing interference and noise in audio signals including speech using generalized sidelobe cancellation (GSC). The embodiments are shown in FIG. 5 and FIG. 6. FIG. 5 shows the system, and FIG. 6 the method. The method can be performed in a processor connected to a memory and input/output interfaces as known in the art.
Voice Activity Detection
In a first stage, a conventional voice activity detector (VAD) 610 can be used to determine when not to perform the GSC update, and instead directly suppress the noise, in noise only segments of speech. We use a modification to the discontinuity in the detection. Because VAD gives a 0/1 indicator, which signals speech muted and active, respectively. Both VAD errors and the speech activity's frequent changes would make the output speech discontinuous and uncomfortable if the detected non-speech parts are fully canceled by setting the non-speech parts to 0. However, long time non-speech parts can be fully cancelled without perceptual discomfort, while a short time non-speech part cannot. The voice activity detection is therefore followed by detecting 620 noise only segments. If there are a sufficient number of such segments 625 in a neighborhood of segments, the noise suppression 627 can be performed. Otherwise, proceed with overlapping speech (OS) detection 630.
Overlapping Speech Detection
In automatic overlapping speech (OS) detection (ASD) 640, there can be an array 501 of two or more microphones, and arbitrary number (N) of audio signal sources. The detection can be frame by frame on the observed data xi 501, i=1, 2 is the microphone index for any pair of microphones in the array. The observed data are obtained by audio signals acquired by the array of microphones.
In particular, the audio signal includes speech from a target source, i.e., a driver. The audio signal can also include noise, e.g., music and radio, interfering speech, etc., speech from sources other than the target.
The mixing model is xi(t)=Σj=1Nhij*sj(t), where t is time, hij is an impulse response from source sj to microphone i, which is a linear convolution. If sk, kε{1, 2, . . . , N} in a frame D is the only active source, then it follows from the mixing model that

h2k*x1(t)−h1k*x2(t)=0 for tεD. 

The elimination by cross-channel cancellation is known in blind channel identification. Based on the observed data in D, a pair of sparse filters ui (i=1, 2) are used to minimize the energy of u2*x1−u1*x2 in the region D. Ideally, u1≈h1k and u2≈h2k. The sparseness of the finite impulse response filters regularizes the solution and improves the accuracy of the method. Filter sparseness is achieved by l1-norm regularization 530.
The resulting convex optimization problem for tεD is




  
    
      
        
          
            
              (
              
                
                  u
                  1
                  *
                
                ,
                
                  u
                  2
                  *
                
              
              )
            
            =
            
              
                arg
                ⁢
                
                  
                
                ⁢
                
                  
                    min
                    
                      (
                      
                        
                          u
                          1
                        
                        ,
                        
                          u
                          2
                        
                      
                      )
                    
                  
                  ⁢
                  
                    
                      1
                      2
                    
                    ⁢
                    
                      
                        
                        
                          
                            
                              u
                              2
                            
                            *
                            
                              x
                              1
                            
                          
                          -
                          
                            
                              u
                              1
                            
                            *
                            
                              x
                              2
                            
                          
                        
                        
                      
                      2
                      2
                    
                  
                
              
              +
              
                
                  
                    η
                    2
                  
                  2
                
                ⁢
                
                  
                    (
                    
                      
                        
                          ∑
                          
                            i
                            =
                            1
                          
                          2
                        
                        ⁢
                        
                          
                            μ
                            i
                          
                          ⁡
                          
                            (
                            1
                            )
                          
                        
                      
                      -
                      1
                    
                    )
                  
                  2
                
              
              +
              
                
                  μ
                  (
                  
                    
                      
                        
                        
                          u
                          1
                        
                        
                      
                      1
                    
                    ⁢
                    
                      u
                      2
                    
                  
                  
                
                1
              
            
          
          )
        
        ,
      
    
    
      
        (
        7
        )
      
    
  




where the second term





  
    
      η
      2
    
    2
  
  ⁢
  
    
      (
      
        
          
            ∑
            
              i
              =
              1
            
            2
          
          ⁢
          
            
              μ
              i
            
            ⁡
            
              (
              1
              )
            
          
        
        -
        1
      
      )
    
    2
  




is to fix scaling and prevent zero (trivial) solution.

Denote the length of D by LD and that of ui by L. The frame size of D can be set as short as 100 ms. As a result, this spatial difference based method solves the problem efficiently, in terms of the data usage, and is different from other OSD methods based on high order statistics of data.
Because the solution ui is l1 regularized by solving Eq. 7, the filter length is effectively truncated to a size less than or equal to the length D supported by the data. In addition, sparseness forces the solution ui to be able to resolve the major spikes of the channel impulse response filters which comprise the relative time delay. In this sense, it helps to be insensitive and accurate under reverberant conditions. In matrix form, the convex objective of Eq. 7 becomes




  
    
      
        
          u
          *
        
        =
        
          
            arg
            ⁢
            
              
            
            ⁢
            
              
                min
                u
              
              ⁢
              
                
                  1
                  2
                
                ⁢
                
                  
                    
                    
                      Au
                      -
                      f
                    
                    
                  
                  2
                  2
                
              
            
          
          +
          
            μ
            ⁢
            
              
                
                u
                
              
              1
            
          
        
      
    
    
      
        (
        8
        )
      
    
  




where u is formed by stacking up u1 and u2; vector f=(0, 0, . . . , 0, η)T with length LD+1; and (LD+1)×2L matrix A (T is transpose) is





  A
  =
  
    
      (
      
        
          
            
              
                x
                1
              
              ⁡
              
                (
                1
                )
              
            
          
          
            
              
                x
                1
              
              ⁡
              
                (
                2
                )
              
            
          
          
            …
          
          
            …
          
          
            
              
                x
                1
              
              ⁡
              
                (
                
                  
                    L
                    D
                  
                  -
                  1
                
                )
              
            
          
          
            
              
                x
                1
              
              ⁡
              
                (
                
                  L
                  D
                
                )
              
            
          
          
            η
          
        
        
          
            
              
            
          
          
            
              
                x
                1
              
              ⁡
              
                (
                1
                )
              
            
          
          
            …
          
          
            …
          
          
            
              
                x
                1
              
              ⁡
              
                (
                
                  
                    L
                    D
                  
                  -
                  2
                
                )
              
            
          
          
            
              
                x
                1
              
              ⁡
              
                (
                
                  
                    L
                    D
                  
                  -
                  1
                
                )
              
            
          
          
            0
          
        
        
          
            
              
            
          
          
            
              
            
          
          
            ⋱
          
          
            
              
            
          
          
            
              
            
          
          
            ⋮
          
          
            ⋮
          
        
        
          
            
              
            
          
          
            
              
            
          
          
            
              
            
          
          
            
              
                x
                1
              
              ⁡
              
                (
                1
                )
              
            
          
          
            …
          
          
            
              
                x
                1
              
              ⁡
              
                (
                
                  
                    L
                    D
                  
                  -
                  L
                  +
                  1
                
                )
              
            
          
          
            0
          
        
        
          
            
              -
              
                
                  x
                  2
                
                ⁡
                
                  (
                  1
                  )
                
              
            
          
          
            
              -
              
                
                  x
                  2
                
                ⁡
                
                  (
                  2
                  )
                
              
            
          
          
            …
          
          
            …
          
          
            
              -
              
                
                  x
                  2
                
                ⁡
                
                  (
                  
                    
                      L
                      D
                    
                    -
                    1
                  
                  )
                
              
            
          
          
            
              -
              
                
                  x
                  2
                
                ⁡
                
                  (
                  
                    L
                    D
                  
                  )
                
              
            
          
          
            η
          
        
        
          
            
              
            
          
          
            
              -
              
                
                  x
                  2
                
                ⁡
                
                  (
                  1
                  )
                
              
            
          
          
            …
          
          
            …
          
          
            
              -
              
                
                  x
                  2
                
                ⁡
                
                  (
                  
                    
                      L
                      D
                    
                    -
                    2
                  
                  )
                
              
            
          
          
            
              -
              
                
                  x
                  2
                
                ⁡
                
                  (
                  
                    
                      L
                      D
                    
                    -
                    1
                  
                  )
                
              
            
          
          
            0
          
        
        
          
            
              
            
          
          
            
              
            
          
          
            ⋱
          
          
            
              
            
          
          
            
              
            
          
          
            ⋮
          
          
            ⋮
          
        
        
          
            
              
            
          
          
            
              
            
          
          
            
              
            
          
          
            
              -
              
                
                  x
                  2
                
                ⁡
                
                  (
                  1
                  )
                
              
            
          
          
            …
          
          
            
              -
              
                
                  x
                  2
                
                ⁡
                
                  (
                  
                    
                      L
                      D
                    
                    -
                    L
                    +
                    1
                  
                  )
                
              
            
          
          
            0
          
        
      
      )
    
    T
  




with the indices of x1 and x2 restricted in D.

In each frame D, whether a SS frame or OS frame, we obtain an output signal ŝD=u2D*x1−u1D*x2, where uiD i=1, 2 are the optimal solution of Eq. 7 solved in D. Then for the whole utterance, xi (i=1 or 2) and ŝD are normalized such that l2 norm is 1. Finally the energy ratio restricted in D between normalized xi and ŝD is calculated for detection as




  
    
      
        
          R
          D
        
        =
        
          
            
              
                
                
                  
                    x
                    i
                  
                  ⁢
                  
                    ❘
                    D
                  
                
                
              
              2
              2
            
            
              
                
                
                  
                    
                      s
                      ^
                    
                    D
                  
                  ⁢
                  
                    ❘
                    D
                  
                
                
              
              2
              2
            
          
          .
        
      
    
    
      
        (
        9
        )
      
    
  




The cross-channel cancellation does not work if the frame D contains multiple sources, i.e., ∥ŝD|D∥2 is not much smaller than ∥xi|D∥2. Hence, the significantly large values of this ratio resulting from the good performance of cross-channel cancellation indicate the non-overlapping speech frames.
Usually, we select the frames with the ratio larger than r0 as the SS frames, where r0 is a threshold value.
FIG. 2 shows OSD on recorded audio samples. Two microphones receive a mixed audio signal from four sources (s1, . . . , s4). The four sources with the parts marked 201 are for single speaker speech.
FIG. 3 and FIG. 4 show the energy ratio of the ground truth, and the energy ratio for detection in each time frame, respectively. The ratio of ground truth in each frame is the energy ratio between the source with the largest energy, and the source with second largest energy. The threshold is selected as 50.
The ratio of the ground truth indicates which frame contains only one active or a significantly dominant source, while the ratio of detection is the output of the process, which is used to determine whether a frame contains speech from a single speaker, or not. The detection ratios agree qualitatively with the ground truth.
FIG. 7 shows pseudo code for our OSD process. The variables and equations are described in detail above.
DOA Estimate
The delay of arrival (DOA) estimation 645 between each of the considered channel and the reference channel is repeated for two purposes. On one hand, the DOA estimate for the single speaker (SS) segments helps to select a period of speech to estimate the spatial difference from a vehicle driver 670 to the microphones, which is used to steer the array before our sparse GSC 660, and generate the BM as described below.
On the other hand, the DOA estimate helps speaker turn detection, see below, to select all the segments where only the driver is speaking 670. These segments are directly sent to the post-processing to surpress 675 interference, without being processed by our sparse GSC.
As used herein, sparsity is not a relative term, but rather a term of art. In signal processing and numerical analysis, a sparse signal or sparse matrix is primarily populated with zeros. In particular, signals are sparse when the signals contain many coefficients close to or equal to zero. Sparsity corresponds to systems that are loosely coupled. When storing and manipulating sparse matrices, it is beneficial and often necessary to use specialized processes and data structures that take advantage of the sparse structure of the matrix, as described herein.
To determine 645 the DOA between the reference channel and any other channel for any given segment, we estimate the DOA as the delay that causes the cross-correlation between the two signals segments to be a maximum. To improve accuracy in the case of reverberation, it is normal practice to use the Generalized Cross Correlation with Phase Transform (GCC-PHAT).
Sparse GSC
The signals discussed in this section (661) are all overlapping speech (OS), i.e., the target speech is interfered by other point audio sources. The observed signals from the microphone array is “steered” at the target using the FBF 510. The FBF is used to enhance a target signal from a selected direction to a target, e.g., the vehicle driver. The direction of the target source, before steering, is typically the angle between the line extending from the center of the microphone array projected through the target source, and a line connecting the microphones. This direction is estimated by the short time average DOA of the single speaker segments, which in some applications can be assumed to be in a restricted angular range. For example, this range might be [20°, 50°], when the microphone array is suspended from the upper windshield area of the car, in front of the driver and passenger seats, and laterally centered along the midline of the car, and angles are measured such that positive angles indicate the driver's side and negative angles indicate the passenger side. Conventional beamforming methods 120 assume that the target signal propagates through a known direct path. However, in applications such as speech acquisition in adverse acoustic environments, e.g., vehicles, the source signal propagates along a direct path, and unknown multi-path, subject to reverberation effects. In such cases, the target signal is often cancelled to some extent by conventional adaptive beamforming methods. This problem is especially serious for microphone arrays in strong reverberant environments, such as vehicles. The performance significantly degrades due to the reverberation effects.
Concerning the existence of reverberation effects, we use a novel channel model in the array processing. The spatial difference of the channel relating target source and two of the array elements is modeled as a pair of sparse filters to account for the reverberation effects.
The blocking matrix (filter) 520 is used to suppress the target signal but enhance the interference and noise. In the reverberant environment, because the signal components at different frequency bins have different response characteristics, the conventional blocking filter cannot block the target signal efficiently. When the target signal leaks into the multi-channel adaptive filter, it results in target signal cancellation.
Therefore, we modify the conventional BM 120 by including temporal information to block all the components of the target signal in our BM 520.
The GCC-PHAT is applied on the segments selected by the OSD, which confirms that these segments are due to a single point source. In the automotive environment, the driver's possible region is approximately [20°,50°]. Therefore, the segments with DOA in this range indicate the driver's speech. For the purpose of the computational efficiency, the OSD stops after the first driver only segment (denoted as D0) is located.
Microphone i, i=1, 2, . . . , M−1 is set as the reference channel, one by one, respectively. This optimization (Eq. 7) is called in D0 for each pair of observed data xi and xi+1, to obtain a pair of sparse filters uii and ui+1i, where the superscript i indicates that ith reference channel is involved in the optimization problem and the subscript indicates the estimated channel index. Eventually,




  
    
      
        
          y
          i
          BM
        
        =
        
          
            
              
                
                  u
                  i
                  i
                
                *
                
                  x
                  
                    i
                    +
                    1
                  
                
              
              -
              
                
                  u
                  
                    i
                    +
                    1
                  
                  i
                
                *
                
                  x
                  i
                
              
            
            ≈
            
              
                
                  h
                  
                    i
                    ,
                    
                      k
                      0
                    
                  
                
                *
                
                  x
                  
                    i
                    +
                    1
                  
                
              
              -
              
                
                  h
                  
                    
                      i
                      +
                      1
                    
                    ,
                    
                      k
                      0
                    
                  
                
                *
                
                  x
                  i
                
              
            
          
          =
          
            
              ∑
              
                j
                ≠
                
                  k
                  0
                
              
              
                
              
            
            ⁢
            
              
                (
                
                  
                    
                      h
                      
                        i
                        ,
                        
                          k
                          0
                        
                      
                    
                    *
                    
                      h
                      
                        
                          i
                          +
                          1
                        
                        ,
                        j
                      
                    
                  
                  -
                  
                    
                      h
                      
                        
                          i
                          +
                          1
                        
                        ,
                        
                          k
                          0
                        
                      
                    
                    *
                    
                      h
                      
                        i
                        ,
                        j
                      
                    
                  
                
                )
              
              *
              
                
                  s
                  j
                
                .
              
            
          
        
      
    
    
      
        (
        10
        )
      
    
  




The source sk0, the only active source from driver in D0, is eliminated in the entire utterance, provided that the acoustic environment does not change so that estimation of real impulse response from target k0 to microphone i and i+1 (hi,k0 and hi+1,k0) during D0 is still valid for the whole utterance.
The blocking filter Wb 520 is constructed as




  
    
      
        
          W
          b
        
        =
        
          
            (
            
              
                
                  
                    u
                    2
                    1
                  
                
                
                  
                    -
                    
                      u
                      1
                      1
                    
                  
                
                
                  0
                
                
                  0
                
              
              
                
                  0
                
                
                  
                    u
                    3
                    2
                  
                
                
                  
                    -
                    
                      u
                      2
                      2
                    
                  
                
                
                  0
                
              
              
                
                  0
                
                
                  0
                
                
                  
                    u
                    4
                    3
                  
                
                
                  
                    -
                    
                      u
                      4
                      3
                    
                  
                
              
            
            )
          
          .
        
      
    
    
      
        (
        11
        )
      
    
  




The output signal yBM of the blocking filter is

yBM=Wb*X.  (12),

where X=(x1; x2; . . . ; xM) is the steered signals and * is a linear convolution.

The output signal of the multichannel adaptive noise cancellation is




  
    
      
        
          y
          =
          
            
              y
              FBF
            
            -
            
              
                ∑
                
                  m
                  =
                  1
                
                
                  M
                  -
                  1
                
              
              ⁢
              
                
              
              ⁢
              
                
                  g
                  m
                
                *
                
                  y
                  m
                  BM
                
              
            
          
        
        ,
      
    
    
      
        (
        13
        )
      
    
  




where ymBM is m-th output of BM, m=1, 2, . . . , M−1.

There is a trade-off between sufficiently determining the filter gm and the computation efficiency. A relatively long filter order, in which the number of coefficients in gm is large, degrades the efficiency of the iteration by the least mean square (LMS), while a relatively short filter gm that is less than the reverberation time of the vehicle (which may be 50 ms to 150 ms), cannot suppress all interference. With long filters of around 150 ms, sparse regularized convex optimization is incorporated and solved efficiently by the split Bregman method, see below.
Sparse regularization can be used to improve the estimation of the filters as follows. The optimal solution g=(g1; g2; . . . ; gM−1) is obtained by minimizing the variance of beamforming output y with sparse regularization on the optimal solution g




  
    
      
        
          
            g
            *
          
          =
          
            
              
                min
                g
              
              ⁢
              
                
                  
                  
                    
                      y
                      FBF
                    
                    -
                    
                      
                        ∑
                        
                          m
                          =
                          1
                        
                        
                          M
                          -
                          1
                        
                      
                      ⁢
                      
                        
                      
                      ⁢
                      
                        
                          g
                          m
                        
                        *
                        
                          y
                          m
                          BM
                        
                      
                    
                  
                  
                
                2
                2
              
            
            +
            
              μ
              ⁢
              
                
                  ∑
                  
                    m
                    =
                    1
                  
                  
                    M
                    -
                    1
                  
                
                ⁢
                
                  
                
                ⁢
                
                  
                    
                    
                      g
                      m
                    
                    
                  
                  1
                
              
            
          
        
        ,
      
    
    
      
        (
        14
        )
      
    
  




which can be rewritten with matrix form as (Eqn. 8) and implemented by the split Bregman method.

Furthermore, the output y of the sparse GSC corrects the reference of noise and interference yBM at the BM stage. For each m, m=1, 2, . . . , M, cross cancellation is applied to ymBM and y by solving




  
    
      
        
          (
          
            
              u
              1
              *
            
            ,
            
              u
              2
              *
            
          
          )
        
        =
        
          
            arg
            ⁢
            
              
                min
                
                  (
                  
                    
                      u
                      1
                    
                    ,
                    
                      u
                      2
                    
                  
                  )
                
              
              ⁢
              
                
                  1
                  2
                
                ⁢
                
                  
                    
                    
                      
                        
                          u
                          2
                        
                        *
                        
                          y
                          m
                          BM
                        
                      
                      -
                      
                        
                          u
                          1
                        
                        *
                        y
                      
                    
                    
                  
                  2
                  2
                
              
            
          
          +
          
            
              
                η
                2
              
              2
            
            ⁢
            
              
                (
                
                  
                    
                      ∑
                      
                        i
                        =
                        1
                      
                      2
                    
                    ⁢
                    
                      
                    
                    ⁢
                    
                      
                        u
                        i
                      
                      ⁡
                      
                        (
                        1
                        )
                      
                    
                  
                  -
                  1
                
                )
              
              2
            
          
          +
          
            
              μ
              ⁡
              
                (
                
                  
                    
                      
                      
                        u
                        1
                      
                      
                    
                    1
                  
                  +
                  
                    
                      
                      
                        u
                        2
                      
                      
                    
                    1
                  
                
                )
              
            
            .
          
        
      
    
    
      
        (
        15
        )
      
    
  




The noise and interference reference ymBM is further updated by

ymBM←u2*ymBM−u1*y.  (16)

The sparse GSC is carried out by this loop updating yBM and y.
FIG. 5 shows our sparse GSC, which includes the FBF 510, the block filters 520, and the sparse regulizers 530, for input 501, and output 502.
It should be noted that the basic method can be performed iteratively until a termination condition is reached. In this embodiment, during each iteration, the filters are further optimized, and the interference and target signal are reestimated accordingly. The termination condition can be convergence of the filter parameters, or a fixed number of iterations.
Speaker Turn Detection
Speaker turn detection (STD) 650 is used on the observed speech data for which different point sources are active in turn. Distinguishing speakers by DOA estimation could be used. However, the difference of the DOAs in a vehicle is sometime not significant, especially when a radio or a music player is on, or a passenger in back seat behind the driver is speaking.
In addition, the noise in the vehicle degrades the performance of the DOA estimation as well. Therefore, we use a more accurate method for STD, by an initial blind speech separation (BSS).
Without loss of generality, M sources are active in turn in the observed data xm, m=1, 2, . . . , M. By calling the BSS method independent vector analysis (IVA), output zm, m=1, 2, . . . , M are generated as the estimate of each point source.
We relax the criterion of convergence, and stopping iteration before sufficient convergence, to accelerate the processing. For each k, k=1, 2, . . . , M, we normalize zk and Σm≠kzm respectively such that l2 norm is 1. The energy ratio between zk and Σm≠kzm (normalized) is calculated frame by frame (frame index j) as




  
    
      
        
          
            R
            k
          
          ⁡
          
            (
            j
            )
          
        
        =
        
          
            
              
              
                
                  z
                  k
                
                ⁢
                
                  |
                  j
                
              
              
            
            2
            2
          
          
            
              
              
                
                  
                    ∑
                    
                      m
                      ≠
                      k
                    
                  
                  ⁢
                  
                    
                  
                  ⁢
                  
                    z
                    m
                  
                
                ⁢
                
                  |
                  j
                
              
              
            
            2
            2
          
        
      
    
    
      
        (
        17
        )
      
    
  




Though the DOA estimation is not accurate enough to segment the speech by speakers. By labeling each frame, it can be used to label zm, m=1, 2, . . . , M as whose speech is dominant in zm. We need to locate the track k0 where the driver's speech is dominant. Finally, the significantly large values in Rk0 indicate driver-only frames. The rest frames are replaced by 0's.
Post-Filtering
Signals sent to post-filtering 665 are further enhanced by an optimally-modified log-spectral amplitude (OM-LSA) speech estimator, which performs well for non-stationary noise environment, such as car. The spectral gain function, which minimizes the mean-square error of the log-spectra, is obtained as a weighted geometric mean of the hypothetical gains associated with the speech presence uncertainty. The noise estimate is an average of previous spectral power values, using a smoothing parameter that is adjusted by the speech presence probability in subbands.
Implementation
The split Bregman method is an efficient tool for solving optimization problems involving total variation, or l1 regularizations. The method solves the unconstrained problem:




  
    
      
        min
        u
      
      ⁢
      
        J
        ⁡
        
          (
          
            Φ
            ⁢
            
              
            
            ⁢
            u
          
          )
        
      
    
    +
    
      H
      ⁡
      
        (
        u
        )
      
    
  
  ,




where J is convex, but not necessarily differentiable such as the l1 norm, H is convex and differentiable, and Φ is a linear operator. The idea of the split Bregman method is to introduce an auxiliary variable d=Φu, and solve the constrained problem





  
    
      
        min
        
          d
          ,
          u
        
      
      ⁢
      
        J
        ⁡
        
          (
          d
          )
        
      
    
    +
    
      H
      ⁡
      
        (
        u
        )
      
    
  
  ,
  
    
      s
      .
      t
      .
      
        
      
      ⁢
      d
    
    =
    
      Φ
      ⁢
      
        
      
      ⁢
      
        u
        .
      
    
  




This problem can be solved by the following iterations:



  
    
      (
      
        
          u
          
            k
            +
            1
          
        
        ,
        
          d
          
            k
            +
            1
          
        
      
      )
    
    =
    
      
        arg
        ⁢
        
          
            min
            
              u
              ,
              d
            
          
          ⁢
          
            J
            ⁡
            
              (
              d
              )
            
          
        
      
      +
      
        H
        ⁡
        
          (
          u
          )
        
      
      -
      
        〈
        
          
            p
            d
            k
          
          ,
          
            d
            -
            
              d
              k
            
          
        
        〉
      
      -
      
        〈
        
          
            p
            u
            k
          
          ,
          
            u
            -
            
              u
              k
            
          
        
        〉
      
      +
      
        
          λ
          2
        
        ⁢
        
          
            
            
              d
              -
              
                Φ
                ⁢
                
                  
                
                ⁢
                u
              
            
            
          
          2
          2
        
      
    
  




  
    
      
    
    ⁢
    
      
        p
        d
        
          k
          +
          1
        
      
      =
      
        
          p
          d
          k
        
        -
        
          λ
          ⁡
          
            (
            
              
                d
                
                  k
                  +
                  1
                
              
              -
              
                Φ
                ⁢
                
                  
                
                ⁢
                
                  u
                  
                    k
                    +
                    1
                  
                
              
            
            )
          
        
      
    
  




  
    
      
    
    ⁢
    
      
        
          p
          u
          
            k
            +
            1
          
        
        =
        
          
            p
            u
            k
          
          -
          
            
              λΦ
              T
            
            ⁡
            
              (
              
                
                  Φ
                  ⁢
                  
                    
                  
                  ⁢
                  
                    u
                    
                      k
                      +
                      1
                    
                  
                
                -
                
                  d
                  
                    k
                    +
                    1
                  
                
              
              )
            
          
        
      
      ,
    
  



where •,• is the inner product. For simplicity, we introduce a new variable bk=pdk/λ, and note that pdk=λbk and puk=−λΦTbk. Then dk+1 and uk+1 can be updated alternatively.

The general split Bregman iteration is



  
    
      d
      
        k
        +
        1
      
    
    =
    
      
        arg
        ⁢
        
          
            min
            d
          
          ⁢
          
            
              1
              λ
            
            ⁢
            
              J
              ⁡
              
                (
                d
                )
              
            
          
        
      
      -
      
        〈
        
          
            b
            k
          
          ,
          
            d
            -
            
              d
              k
            
          
        
        〉
      
      +
      
        
          1
          2
        
        ⁢
        
          
            
            
              d
              -
              
                Φ
                ⁢
                
                  
                
                ⁢
                
                  u
                  k
                
              
            
            
          
          2
          2
        
      
    
  




  
    
      u
      
        k
        +
        1
      
    
    =
    
      
        arg
        ⁢
        
          
            min
            u
          
          ⁢
          
            
              1
              λ
            
            ⁢
            
              H
              ⁡
              
                (
                u
                )
              
            
          
        
      
      +
      
        〈
        
          
            b
            k
          
          ,
          
            Φ
            ⁡
            
              (
              
                u
                -
                
                  u
                  k
                
              
              )
            
          
        
        〉
      
      +
      
        
          1
          2
        
        ⁢
        
          
            
            
              
                d
                
                  k
                  +
                  1
                
              
              -
              
                Φ
                ⁢
                
                  
                
                ⁢
                u
              
            
            
          
          2
          2
        
      
    
  




  
    
      b
      
        k
        +
        1
      
    
    =
    
      
        b
        k
      
      -
      
        (
        
          
            d
            
              k
              +
              1
            
          
          -
          
            Φ
            ⁢
            
              
            
            ⁢
            
              u
              
                k
                +
                1
              
            
          
        
        )
      
    
  



In the case of (Eqn. 8), J(u)=μ∥u∥1, Φ=I, and




  
    H
    ⁡
    
      (
      u
      )
    
  
  =
  
    
      1
      2
    
    ⁢
    
      
        
          
          
            Au
            -
            f
          
          
        
        2
        2
      
      .
    
  




The iterations are





  
    
      
        
          d
          
            k
            +
            1
          
        
        =
        
          
            arg
            ⁢
            
              
                min
                d
              
              ⁢
              
                
                  μ
                  λ
                
                ⁢
                
                  
                    
                    d
                    
                  
                  1
                
              
            
          
          -
          
            〈
            
              
                b
                k
              
              ,
              
                d
                -
                
                  d
                  k
                
              
            
            〉
          
          +
          
            
              1
              2
            
            ⁢
            
              
                
                
                  d
                  -
                  
                    u
                    k
                  
                
                
              
              2
              2
            
          
        
      
    
    
      
        (
        18
        )
      
    
  
  
    
      
        
          u
          
            k
            +
            1
          
        
        =
        
          
            arg
            ⁢
            
              
                min
                u
              
              ⁢
              
                
                  1
                  
                    2
                    ⁢
                    λ
                  
                
                ⁢
                
                  
                    
                    
                      Au
                      -
                      f
                    
                    
                  
                  2
                  2
                
              
            
          
          +
          
            〈
            
              
                b
                k
              
              ,
              
                u
                -
                
                  u
                  k
                
              
            
            〉
          
          +
          
            
              1
              2
            
            ⁢
            
              
                
                
                  
                    d
                    
                      k
                      +
                      1
                    
                  
                  -
                  u
                
                
              
              2
              2
            
          
        
      
    
    
      
        (
        19
        )
      
    
  
  
    
      
        
          b
          
            k
            +
            1
          
        
        =
        
          
            b
            k
          
          -
          
            
              (
              
                
                  d
                  
                    k
                    +
                    1
                  
                
                -
                
                  u
                  
                    k
                    +
                    1
                  
                
              
              )
            
            .
          
        
      
    
    
      
        (
        20
        )
      
    
  




Explicitly solving Eqns. 18-19 yields a process



  
    
      Initialize
      ⁢
      
        
      
      ⁢
      
        u
        0
      
    
    =
    
      
        d
        0
      
      =
      
        
          b
          0
        
        =
        0
      
    
  




  
    
      
        While
        ⁢
        
          
          
            
              u
              
                k
                +
                1
              
            
            -
            
              u
              k
            
          
          
        
        ⁢
        
          2
          /
          
            
              
              
                u
                
                  k
                  +
                  1
                
              
              
            
            2
          
        
      
      >
      
        
          ɛ
          ⁢
          
            

          
          (
          1
          )
        
        ⁢
        
          
        
        ⁢
        
          d
          
            k
            +
            1
          
        
      
    
    =
    
      shrink
      ⁡
      
        (
        
          
            
              u
              k
            
            +
            
              b
              k
            
          
          ,
          
            μ
            λ
          
        
        )
      
    
  




  
    
      
        (
        2
        )
      
      ⁢
      
        
      
      ⁢
      
        u
        
          k
          +
          1
        
      
    
    =
    
      
        
          (
          
            
              λ
              ⁢
              
                
              
              ⁢
              I
            
            +
            
              
                A
                T
              
              ⁢
              A
            
          
          )
        
        
          -
          1
        
      
      ⁢
      
        (
        
          
            
              A
              T
            
            ⁢
            f
          
          +
          
            λ
            ⁡
            
              (
              
                
                  d
                  
                    k
                    +
                    1
                  
                
                -
                
                  b
                  k
                
              
              )
            
          
        
        )
      
    
  




  
    
      
        (
        3
        )
      
      ⁢
      
        
      
      ⁢
      
        b
        
          k
          +
          1
        
      
    
    =
    
      
        b
        k
      
      -
      
        d
        
          k
          +
          1
        
      
      +
      
        u
        
          k
          +
          1
        
      
    
  




  
    end
    ⁢
    
      
    
    ⁢
    While
  



The function “shrink” is a soft threshold function defined by shrink(v,t)=(τt(v1), τt(v2), . . . , τt(vNL)) with τt(x)=sign(x)max{|x|−t,0}. The matrix A is fixed, and we can precalculate (λI+ATA)−1, then the iterations only involve matrix multiplication.
Part EFFECT OF THE INVENTION: The invention provides a system and method for generalized sidelobe cancellation (GSC). The method is particularly suited to complex audio environments, such as the interior of vehicles. There, interferences, such as speech from a codriver, rear-seat passengers, music or radio, engine and wind noise, can significantly degrade performance of an automatic speech recognition system.
The invention can be used with speech-based command interfaces in vehicles. Applications include automatic dialog system for hands-free phone calls, as well as more advanced features, such as navigation systems.
The issue of interference is addressed with adaptive interference cancellation techniques, i.e., generalized sidelobe cancellation (GSC), using sparse regularization techniques to improve its robustness.
The embodiments distinguishes noise only segments (NO), single speaker segments (SS), and overlapping speech segments (OS), and then apply different sub-approaches to suppress diffuse noises and other interferences within different types of segments.
A first stage is carried out by overlapping speech detection (OSD) and a voice activity detection, while the second stage is achieved by a blind speech separation (BSS) based speaker turn detection, direction of arrival estimation, sparse GSC and post-processing in the form of log-spectral amplitude estimation.
Although the invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention. Therefore, it is the object of the appended claims to cover all such variations and modifications as come within the true spirit and scope of the invention.


{'bibliographic_information': "{'country': 'US', 'doc-number': '13690201', 'kind': 'B2', 'date': '20121130', 'extension': '328', 'invention_title': 'Method and system for reducing interference and noise in speech signals'}", 'source_file': './grant/2015/ipg150602.xml', 'source_data': 'USPTO-Grant'}