

\section{Conclusions}




We demonstrate the first deep learning  method for real-time and streaming target sound extraction. 
Future work includes the use of more constrained computing platforms,  larger datasets with more classes, and multiple microphones.  Our  Waveformer architecture may be applicable to other   acoustic  applications like source separation and directional hearing, which deserves further exploration



\section{Introduction}
\label{sec:intro}
Humans are exceptionally adept at attending their auditory focus to specific sounds even in a noisy environment~\cite{2020arXiv200605712O}. Recent works that aim to create a computational equivalent of this human capability  formulate this problem as target sound extraction~\cite{2020arXiv200605712O,delcroix2021few,delcroix2022soundbeam}.  The goal   is to extract sound signals of interest from a mixture of various overlapping sounds, given clues which provide information about the target sound class such as embeddings of a one-hot label~\cite{2020arXiv200605712O}, audio clips~\cite{delcroix2022soundbeam,gfeller2021one}, and images~\cite{gao2019co,xu2019recursive}. Streaming target sound extraction  could enable real-time  intelligent acoustic  applications for  headphones, hearing aids and telephony  by filtering out undesired sounds from the environment (e.g., traffic) and presenting only sounds of interest to the user (e.g., sirens).


Recent works on target sound extraction  have shown promising performance even for mixtures containing a large  number of  sound classes~\cite{2020arXiv200605712O}. However, none of these  prior works   demonstrate real-time  streaming capabilities. In particular, the prior works for this task  are based on non-streaming models and designed for offline processing, where the neural network has access to a large block 
($\ge 1$~s) of audio samples~\cite{2020arXiv200605712O}. In contrast, real-time  streaming applications impose significant {algorithmic and} computational  constraints, requiring networks to     operate on small blocks 
($\le 10$~ms) with a limited number of lookahead samples  for each block. All these factors can significantly degrade  the performance~\cite{luo2019conv}.

In this paper, we present the first deep learning method to perform  target sound extraction in a streaming manner.  Fig.~\ref{fig:main_fig} shows  Waveformer, our encoder-decoder architecture where the encoder is a stack of dilated causal convolution (DCC) layers and the decoder is a transformer decoder \cite{https://doi.org/10.48550/arxiv.1706.03762}.\footnote{We call  our network, Waveformer, since it uses an hybrid architecture with the causal convolution layers,  common in WaveNet ~\cite{https://doi.org/10.48550/arxiv.1609.03499} based architectures, as the encoder and a transformer as the decoder.}  {Our intuition is that much of the complexity in  prior  models comes with processing large receptive fields, especially at high sampling rates. For example, recent transformer based architectures proposed for speech separation \cite{subakan2022resource, luo2022tiny} implement chunk-based processing, where each chunk independently attends to all the chunks in the receptive field. Thus, to achieve a receptive field of length $R$, for each chunk, these models have an $\mathcal{O}(R)$ computational complexity. Instead, since DCC layers have a complexity of $\mathcal{O} (\log R)$ for achieving the same amount of receptive field (\xref{sec:encoder}), we use a stack of DCC layers as the encoder that processes the receptive field. We then use the decoder layer of the transformer architecture \cite{https://doi.org/10.48550/arxiv.1706.03762} as our model's decoder. We leverage the \textit{memory--target} paradigm of the transformer decoder, where the query-conditioned encoder's output is considered as the transformer decoder's `target' and the unconditioned encoder's output is considered as the transformer decoder's `memory'. Using self-attention on the `target' followed by cross-attention between the  `target' and `memory,' the decoder generates the mask for extracting the specified target sound to produce the output signal.}



To evaluate our network architecture, we implement a causal version of Conv-TasNet and a streaming  version of ReSepformer~\cite{subakan2022resource} for the task of streaming target sound extraction. Evaluations show that our hybrid network architecture achieves state-of-the-art performance for this task. 
Further, the smallest and largest versions of our model have real-time factors (RTFs) of  0.66 and 0.94, respectively,  on a consumer-grade CPU, demonstrating the real-time target sound extraction capability.
This is shown to be faster than the two representative prior models while outperforming them in terms of the output signal quality. 


\section{Waveformer Architecture}

 We process  individual audio chunks of duration $\tau$ seconds. For streaming, we need to operate at chunk level: an output chunk can  depend on the current and past chunks. Thus, streaming models have an intrinsic latency equal to the duration of a single chunk.  In real-time practical systems, it is desirable that this   latency is on the order of 10 ms \cite{7952149}.  Fig.~\ref{fig:main_fig} shows our proposed time-domain model architecture, which employs an encoder-decoder based mask generation network to generate an element-wise multiplicative mask in the latent space. Let $x_k \in R^{S}$ denote the current input audio chunk, where $S = \tau F_s$ is the   number of audio samples included in the current chunk with a sampling rate of $F_s$. In the first step, a 1D-convolution layer with stride $L$ and kernel size $3L$ is applied to the input  audio chunk $x_k$ to obtain the latent space representation, $y_k \in R^{E \times K}$, where $E$ is the latent space feature dimensions and $K = \frac{S}{L}$ is the feature sequence length in the latent space. {Setting the kernel size to $3L$ and  stride  to $L$ requires an overlap of $L$ samples with the previous and the future chunk, resulting in a lookahead of $L$ samples. In our experiments, we set $L = 32$ samples at 44.1 kHz. This results in a lookahead of around  $0.73$~ms, which is negligible.} Given a one-hot or multi-hot query vector $q \in \{0, 1\}^{N_c}$, where $N_c$ is the total number of classes, streaming target sound extraction is achieved by computing feature masks  $m_k \in R^{E \times K}$. With the mask generation network and element-wise multiplication denoted as $\mathcal{M}$ and $\odot$, respectively, the  target sound signal, $\hat{x}_k \in R ^ S$, is computed  as:
\begin{align}
    y_k =& \text{Conv1d}(x_k), ~~~
    m_k = \mathcal{M}(y_k\ |\ y_{k-1}, .., y_2, y_1, q) \notag\\
    \hat{x}_k =& \text{ConvTranspose1d}(y_k \odot m_k).\notag
\end{align}

\subsection{Dilated causal convolution encoder}\label{sec:encoder}
Our encoder is a stack of dilated causal convolution (DCC) layers \cite{https://doi.org/10.48550/arxiv.1609.03499}, and the decoder is a transformer network \cite{https://doi.org/10.48550/arxiv.1706.03762}. The motivation for such an architecture  is that the encoder computes a \emph{contextful} representation of the input chunk, considering the previous chunks up to a certain receptive field, and the decoder conditions the encoder output with the query vector to estimate the target mask. While recent  transformer  models for speech separation \cite{subakan2021attention, subakan2022resource} have demonstrated  performance gains over convolution based methods \cite{luo2019conv}, the latter have generally been more computationally efficient.

We attribute this efficiency gap  to the difference in the way existing transformer models  process the receptive field compared to convolution based architectures. To achieve a receptive field of length $R$, given the chunk based processing in existing transformer architectures, each chunk individually attends to all previous chunks in the receptive field resulting in $\mathcal{O}(R)$ complexity. In contrast, convolution based models  \cite{https://doi.org/10.48550/arxiv.1609.03499, luo2019conv} using a stack of $M$ DCC layers with kernel size $P$ and exponentially scaling dilation factors $\{2^0, 2^1, 2^2, ..., 2^{M-1}\}$ have a receptive field of $(P - 1) \cdot (2^M - 1)$. Its complexity is $\mathcal{O}(PM)$. With  a small kernel size $P$, the computational complexity of the stacked DCC layers  is $\mathcal{O}(P \cdot \log~(1 + \frac{R}{P - 1}))$ $\sim$ $\mathcal{O} (\log R)$ for it to have a receptive field of length $R$. 

 We use 10  DCC layers with a kernel size of 3 and dilation factors $\{2^0, 2^1, 2^2, ..., 2^{10-1}\}$ in our encoder, resulting in a receptive field of $(3 - 1) \cdot (2 ^ {10} - 1) = 2046 $ samples in the latent space. With the initial input convolution stride $L$ set to  0.73~ms, our encoder's receptive field  is $\approx 1.5 s$. Fig.~\ref{fig:main_fig} (b) shows an  encoding of an input chunk of length 4. For chunk based streaming inference, the encoder maintains a context buffer for each DCC layer. This context is initially computed from the $1s$ receptive field, and then updated dynamically after encoding each subsequent chunk. For encoding a chunk, each DCC layer is fed with the output chunk of the previous layer, left padded with the context of length twice the layer's dilation. After encoding a chunk, the context is updated with the rightmost elements of the padded input for it to be used in encoding the next chunk. For each input chunk $y_k$, the DCC encoder computes an encoded representation,  $e_k \in R^{E \times K}$. 
 
 
\begin{table}
\label{single_target}
\caption{Performance  for  the single-target extraction task. In our model, $E$ and $D$ correspond to encoder and decoder dimensionalities, respectively. RTF is the real-time factor for a consumer  CPU.}
\vspace{-.8em}
\centering
\begin{tabular}{lccc}
\toprule
Model                   & Model size & RTF  & SI-SNRi  \\
\midrule
Conv-TasNet              & 4.57M      & 1.34 & 6.14    \\
ReSepformer             & 13.24M     & 1.60 & {7.26}    \\
Ours ($E = 256;~D = 128$) & 1.10M      & 0.66 & 9.02    \\
Ours ($E = 256;~D = 256$) & 1.69M      & 0.75 & 9.40    \\
Ours ($E = 512;~D = 128$) & 3.29M      & 0.88 & 9.26    \\
Ours ($E = 512;~D = 256$) & 3.88M      & 0.94 & \textbf{9.43}    \\
\bottomrule
\end{tabular}
\label{table:single_target}
\vskip -0.15in
\end{table}


\subsection{Query-conditioned transformer decoder}
\label{sec:decoder}

To get the mask, the encoded representation computed above must be conditioned with the query, $q$. To this end, we first compute an embedding, $l \in R^{E \times 1}$, corresponding to $q$. This is achieved by using an embedding layer comprising three 512-dimensional feed-forward sub-layers with an $N_C$-dimensional input and  an $E$-dimensional output. Our transformer decoder conditions the encoded chunk $e_k$ with the query embedding $l$ and derives the mask as follows.

Fig.~\ref{fig:main_fig} (c) shows our decoder architecture. 
First, we perform multiplicative query integration~\cite{2020arXiv200605712O, delcroix2022soundbeam} to compute the conditioned representation: ${e_k}^\prime = e_k \odot l$. Since transformers are more computationally expensive with higher dimensionality, we first project the encoded representations, ${e_k}^\prime$ and $e_k$, to the decoder dimensions $D \le E$ with $1\times1$ convolution. This results in projected encoded representations,  ${pe}_k, {{pe}_k}^\prime \in R^{D \times K}$. The decoded representations are then computed by passing ${{pe}_k, {pe}_k}^\prime$ to the transformer decoder layer's self-attention and cross-attention blocks, respectively, to obtain target mask ${pm}_k \in R^{D \times K}$ in the projected decoder space. It is then projected back to the encoder dimensions with another $1\times1$ convolution layer to obtain  ${m_k}^\prime \in R^{E \times K}$. Since the bottleneck caused by the projection layers might affect the flow of gradients, 
as depicted in the diagram, we use a skip connection immediately after the multiplicative query integration to the output of the projection layer to compute the final mask: $m_k = {m_k}^\prime + {e_k}^\prime$.

{Within the decoder, we use the    chunk-based streaming attention scheme proposed in \cite{9413535}. As shown in  Fig.~\ref{fig:main_fig}(c), for decoding the current chunk, $e_k$, the transformer decoder only  attends to the samples in the current chunk, $e_k$, and one previous chunk,  $e_{k-1}$. This ensures that the input length to the transformer decoder is fixed at $2K$ (current chunk + one previous chunk) and prevents the inference time from growing as the input audio length increases.}
\section{Method}
For an unnoticeable latency, real-time audio systems require an algorithmic latency less than to 10 ms \cite{7952149}. Targeting such systems, we look at the target sound extraction problem in the context of streaming. While a model's causality is usually defined at sample level, in a streaming application, causality is defined at chunk level: an output chunk can only depend on the current chunk and past chunks. As a result, streaming models have an intrinsic latency equal to the duration of a single chunk. In this work, we propose a deep neural network architecture for streaming target sound extraction that operates on individual audio chunks of arbitrary duration $\tau$, where $\tau$ can be set to the maximum algorithmic latency allowed by the system.

\subsection{Streaming target sound extraction}
\label{sec:stream}
As shown in the Figure \ref{fig:top_level}, we propose a Tasnet \cite{https://doi.org/10.48550/arxiv.1711.00541} style time domain architecture that employs a mask generation network to generate an element-wise multiplicative mask in the latent space. Let $x_k \in R^{S}$ be the input audio chunk, where $S = \tau F_s$ is the chunk length in number of audio samples for a sampling rate $F_s$. In the first step, a 1d-convolution layer with stride $L$ is applied the time domain audio chunk $x_k$ to obtain the latent space representation of the input audio chunk, $y_k \in R^{E \times K}$, where $E$ is the latent space feature dimensions and $K = \frac{S}{L}$ is the chunk length in the latent space. Given a one-hot or multi-hot query vector $q^T \in \{0, 1\}^{N_c}$, where $N_c$ is the total number of classes, the problem of streaming target sound extraction can be formulating as computing the mask $m_k \in R^{E \times K}$. With the mask generation network denoted as $\mathcal{M}$, $\forall k \in \{1, 2, 3, ...\}$ estimated target audio chunk $\hat{x}_k \in R ^ S$ is computed in a streaming fashion as:
\begin{gather*}
    y_k = Conv1d(x_k) \\
    m_k = \mathcal{M}(y_k\ |\ y_{k-1}, .., y_2, y_1, q^T) \\
    \hat{x}_k = ConvTranspose1d(y_k \odot m_k)
\end{gather*}

\subsection{Efficient processing of the receptive field using dilated causal convolutions}
We employ an encoder-decoder architecture for the mask generation. The encoder is a stack of dilated causal convolution (DCC) layers \cite{https://doi.org/10.48550/arxiv.1609.03499} and the decoder is a transformer decoder \cite{https://doi.org/10.48550/arxiv.1706.03762}. The motivation for such an architecture for this problem is that, the encoder computes a \emph{contextful} representation of the input chunk, considering the previous chunks up to a certain receptive filed, and the decoder conditions the encoder output with the query vector to estimate the target mask. Recent works on transformer based models for speech separation \cite{subakan2021attention, subakan2022resource} have demonstrated the performance gains over convolution based methods \cite{luo2019conv}, but convolution based method have generally been more computationally efficient.

We attribute this gap in efficiency to the difference in the way existing transformer based architectures process the receptive filed compared to convolution based architectures. To achieve a receptive field of length $R$, owing to chunk based processing in existing transformer architectures, each chunk individually attends to all previous chunks in the receptive field resulting in $\mathcal{O}(R)$ complexity. Convolution based architectures \cite{https://doi.org/10.48550/arxiv.1609.03499, luo2019conv} implementing a stack of $M$ DCC layers with a kernel size $P$, have a receptive field equal to $(P - 1)\ 2^{M - 1}$, however, the number of operations has the complexity $\mathcal{O}(PM)$. Writing in terms of the receptive field $R$, the computational complexity of the stacked DCC layers to achieve a receptive filed of length $R$ is $\mathcal{O}(1 + \log \frac{R}{P - 1})$ = $\mathcal{O} (\log R)$, as the kernel size $P$ is usually a small constant $\geq 1$.

Hence, in our implementation, we use a stack of DCC layers as the encoder of our mask generation network. We use 10 layers and kernel size equal to 3 in our encoder, resulting a receptive field of $(3 - 1)\ .\ 2 ^ {(10 - 1)} = 1024$ samples in the latent space. With the initial input convolution stride $L$ set to be equivalent to 1 ms, receptive field of our encoder is $\approx1s$. In the figure \ref{fig:enc}, we show an example encoding of an input chunk of length 4. For chunk based streaming inference, the encoder maintains a context buffer for each DCC layer. This context is initially computed from the $\approx1s$ receptive field, and then updated dynamically after encoding each subsequent chunk. For encoding a chunk, each DCC layer is fed with the output chunk of the previous layer, left padded with the context of length twice the layer's dilation. After encoding a chunk, the context is updated with the rightmost elements of the padded input for it be used in encoding the next chunk. For each input chunk $y_k$ (output of the initial $Conv1d$ block), DCC encoder computes an encoded representation $e_k \in R^{E \times K}$. 

\subsection{Query conditioned decoding with linear attention}
Encoder output only depends on the current input chunk and the encoder buffer computed from the receptive field. To obtain the final mask, this encoded representation has to be conditioned with the query vector $q^T$. We first compute an embedding, $l \in R^{E \times 1}$, corresponding to $q^T$ using an embedding layer. The embedding layer comprises of three feed-forward sub-layers, with the hidden dimension equal to 512, and the input \& output dimensions equal to $E$. The decoder conditions the encoded chunk $e_k$ with the query embedding $l$. Motivated by the success of the transformer based architectures for speech processing tasks \cite{https://doi.org/10.48550/arxiv.2010.13154, 9413535}, we employ a transformer decoder layer for estimating the target mask, shown in \ref{fig:dec}.

Similar to \cite{2020arXiv200605712O, delcroix2022soundbeam}, we perform multiplicative query integration to compute conditioned representation: ${e_k}^\prime = e_k \odot l$. Since transformers are more computationally expensive at higher dimensionality, we first project the encoded representations, ${e_k}^\prime$ and $e_k$ to decoder dimensions $D \le E$, using 1x1 convolution layers. This results in projected encoded representations ${pe}_k, {{pe}_k}^\prime \in R^{D \times K}$. The decoded representations are then computed by passing ${{pe}_k, {pe}_k}^\prime$ to the transformer decoder layer's self-attention and cross-attention blocks, respectively, to obtain the target mask in the projected decoder space, ${pm}_k \in R^{D \times K}$. The target mask is then projected back to encoder dimensions using another 1x1 convolution layer to obtain the ${m_k}^\prime \in R^{E \times K}$. All projections include a ReLU non-linearity. Owing to this non-linearity, we observed that the projection layers affect the gradient flow to the embedding layer. To circumvent this, we add a skip connection immediately after the multiplicative query integration to the output of the projection layer to compute the final mask: $m_k = {m_k}^\prime + {e_k}^\prime$. The mask $m_k$ is then used to compute extract the target sound, as described in \ref{sec:stream}.
\section{Related work}
{\bf Universal sound separation.} The task here is to decompose a mixture of arbitrary sound types into their component sounds, regardless of the number of sounds in the mixture~\cite{kavalerov2019universal}. This  becomes increasingly challenging as the number of possible sound types in the mixture increases.  Several networks have been proposed for this task including convolutional long short-term memory networks~\cite{kavalerov2019universal}, time-dilated convolution networks~\cite{kavalerov2019universal} based on Conv-TasNet~\cite{luo2019conv}, and transformer networks~\cite{zadeh2019wildmix}. Prior work also  proposed the use of embeddings learnt by a sound classifier trained on a large sound ontology~\cite{gemmeke2017audio} for conditioning a separation network.

\vskip 0.05in\noindent {\bf Target sound extraction.} This aims at 
separating one or a limited number of sounds of interest from a mixture by conditioning a source-separation model with query information or clues about the target sound. This approach can circumvent the challenge of  universal sound separation struggling to deal with a mixture of a large number of sounds. 
The clues may be provided as an embedding of an audio clip~\cite{delcroix2022soundbeam,gfeller2021one}, an image~\cite{gao2019co,xu2019recursive}, natural language text~\cite{kilgour2022text,liu2022separate}, onomatopoeic words~\cite{okamoto2022environmental}, or a one-hot sound label vector~\cite{2020arXiv200605712O}. The prior works~\cite{kong2020source,chen2022zero} have also evaluated the use of a sound event detector to detect the time when the target sound occurs in a mixture. Although, these works are often motivated for practical usage~\cite{liu2022separate,2020arXiv200605712O}, none of them use streaming models. In contrast, we design the first streaming network for target sound extraction  using attention.

\vskip 0.05in\noindent {\bf Speech-specific networks.}
Prior work  has also  focused on speech enhancement~\cite{wisdom2019differentiable,eskimez2022personalized,chatterjee2022clearbuds},  speech separation~\cite{luo2019conv,luo2020dual,huang2014deep,hershey2016deep,isik2016single,yu2017permutation,wang2018voicefilter} and speech selection using clues provided to the network~\cite{tzinis2022heterogeneous,ephrat2018looking,directional}. For  speech enhancement, neural networks  have been proposed~\cite{eskimez2022personalized} to realize  real-time operation.  {Recently, efficient transformer based architectures have been proposed for speech recognition and separation tasks ~\cite{subakan2021attention,luo2022tiny,https://doi.org/10.48550/arxiv.2005.08100}. These method either use standard transformer blocks \cite{subakan2022resource} or convolution augmented transformer blocks \cite{luo2022tiny, https://doi.org/10.48550/arxiv.2005.08100}. In contrast, we use DCC layers to process the receptive field and a  transformer decoder for generating the target mask for  sound extraction. Recent work on ReSepformer~\cite{subakan2022resource} proposed a causal mode for their transformer method, which  we use  for  our baseline comparisons.}
\section{Related work}
{\bf Universal sound separation} Universal sound separation~\cite{kavalerov2019universal} is the task of separating a mixture of arbitrary sound types into their component sounds, regardless of the number of sounds in the mixture. This task becomes increasingly challenging as the number of possible sound types in the mixture increases. Prior work in this area have made progress by constructing an open dataset with mixtures of sounds drawn from hundreds of different classes~\cite{wisdom2021s}. Several network architectures have been proposed for the task including convolutional long short-term memory networks~\cite{kavalerov2019universal}, time-dilated convolution networks~\cite{kavalerov2019universal} based on ConvTas-Net~\cite{luo2019conv}, and Transformer networks~\cite{zadeh2019wildmix}. Further work proposed the use of embeddings learned by a sound classifier trained on a large ontology of sounds~\cite{gemmeke2017audio} to condition a separation network with additional information. Unlike these works, our work is focused on the related problem of extracting target sound classes from a mixture.

\noindent {\bf Target sound extraction.} Target source extraction refers to the task of separating one or more sounds of interest from a mixture by conditioning a source-separation model with query information or clues about the target like an embedding of an audio clip~\cite{delcroix2022soundbeam}, an image~\cite{gao2019co,xu2019recursive}, natural language text~\cite{kilgour2022text,liu2022separate}, onomatopoeic words~\cite{okamoto2022environmental}, or a one-hot sound label vector~\cite{2020arXiv200605712O}. Prior work~\cite{kong2020source,chen2022zero} has also evaluated the use of a sound event detector to detect the time when a target sound occurs in a mixture. Multiple modalities like natural language and sound samples have also been used in combination to exploit common information between them~\cite{kilgour2022text}. Although, these works are often motivated as being useful for real-time applications~\cite{liu2022separate,2020arXiv200605712O}, none of them are causal. In contrast, we design the first causal network for real-time target sound extraction of diverse sound events. 

\noindent {\bf Speech networks.}
Prior work in the domain of speech has focused on speech enhancement~\cite{wisdom2019differentiable,eskimez2022personalized,chatterjee2022clearbuds}, which is the separation of relatively structured sounds of speech from non-speech interference, speech separation~\cite{luo2019conv,luo2020dual,huang2014deep,hershey2016deep,isik2016single,yu2017permutation,wang2018voicefilter} which is the separation of all voices in a mixture, and speech selection which is the extraction of a target speaker from a mixture given clues provided to the network~\cite{tzinis2022heterogeneous,ephrat2018looking}. For the task of speech enhancement, causal networks based on a complex convolution recurrent network and a deep convolution attention U-Net have been proposed~\cite{eskimez2022personalized} to achieve real-time performance. For speech separation, Transformer networks have recently been used to achieve state-of-the-art performance due to its ability to learn short and long-term dependencies~\cite{subakan2021attention,luo2022tiny}. However, the use of Transformer networks for the task of target sound extraction remains under-explored. In this work we present the first causal Transformer model for real-time target sound extraction.
\section{Experiments and results}





{\bf Dataset.} We use a synthetic sound mixture dataset created from the  FSD Kaggle 2018 dataset~\cite{Fonseca2018_DCASE}. FSD Kaggle 2018 is a set of sound event and class label pairs, with 41 different sound classes, which are a subset of the Audioset ontology \cite{gemmeke2017audio}. Our synthetic dataset consists of 50k training samples, 5k validation samples and 10k test samples. Sound mixtures are created using the Scaper toolkit \cite{8170052}, each with 3-5 foreground sounds randomly sampled from the FSD Kaggle 2018 dataset, and a background sound randomly sampled from the TAU Urban Acoustic Scenes 2019 dataset~\cite{Mesaros2018_DCASE}. Foreground sound classes are randomly sampled without replacement so that each sample has 3-5 unique classes. We construct the sound mixtures by sampling 3-5s crops from each foreground sound and then pasting them on a 6s background sound. The SNRs of the foreground sounds are randomly chosen between 15 and 25 dB, relative to the background sound. Our training and validation data are sampled from the development splits of FSD Kaggle 2018 and TAU Urban Acoustic Scenes 2019, while our test samples are  from the test splits. {From each mixture sample, 3 foreground sounds are randomly selected  to be the target sounds. Only one of those 3 foreground sounds is used for the single-target extraction task, while others are used for the multi-target extraction task. The choices of the target foreground sounds are fixed after  generating the  validation and test sets, to ensure evaluations are reproducible. During training, however, the choices of the target foreground sounds in the training set are randomized. Since we mainly consider human listening applications for streaming target sound extraction, we run  our experiments at a 44.1 kHz  sampling rate  to cover the full audible range.}





\vskip 0.05in\noindent{\bf Evaluation setup.} Prior works \cite{2020arXiv200605712O, delcroix2022soundbeam} show that Conv-TasNet,   originally proposed for speech separation, can also be used for target sound extraction. Further,   ReSepformer proposes an efficient transformer architecture for speech separation that allows a streaming inference. Here, we compare the performance of our architecture  with the causal or streaming implementations of Conv-TasNet and ReSepformer as described in the original papers~\cite{luo2019conv,subakan2022resource} for the  target sound extraction task. 


For all the models, we set the stride of the initial convolution, $L$, to 32, which is  about $0.73$ ms at 44.1 kHz. We train multiple configurations of our model with different encoder and decoder dimensions. We fix the number of DCC layers to 10, number of transformer layers to 1 and chunk length, $K$, to 13. This chunk length corresponds to 416 samples in the time domain, or a chunk duration of 9.43 ms.   
For Conv-TasNet, we follow the configuration used in \cite{2020arXiv200605712O} except for the number of repeats, which we set to 2. This ensures that the runtime of the Conv-TasNet baseline is not too large compared with that of our model's largest configuration. For the ReSepformer baseline, we set the model dimensionality to 512, number of blocks to 2, number of transformer layers to 2 and chunk size to 13 (9.43 ms). We perform label integration after the first transformer block, as we found that to be perform better than integrating it  at the beginning.

\vskip 0.05in\noindent{\bf Loss function and training hyper-parameters.} We use a linear combination of 90\% signal-to-noise-ratio (SNR) and 10\% scale-invariant-signal-to-noise-ratio (SI-SNR) \cite{https://doi.org/10.48550/arxiv.1811.02508} as the loss function for training. While a scale dependent loss, such as SNR loss, is essential for preserving the original amplitude of the desired sounds in the mixture, we observed that adding a fraction of the SI-SNR helps stabilize the training process especially for causal models. We set the initial learning rate to 5e-4 for our models and Conv-TasNet, and to 1.5e-4 for the ReSepformer. We use the  \verb|ReduceLROnPlateau| learning rate scheduler to scale the learning rate by 0.1, if there is no improvement  in the validation SI-SNR for more than 5 epochs. We use SI-SNR improvement (SI-SNRi) as the validation and test metric. We train the models for 100 epochs. We pick the  weights after the epoch that resulted in the best validation SI-SNRi.


\begin{figure}[t!]
\centering
\begin{subfigure}[l]{\linewidth}
    \includegraphics[width=\linewidth]{Figs/qual-mixed.png}
    \caption{Input mixture}
    \label{fig:qual_mixture}
\end{subfigure}
\centering
\begin{subfigure}[l]{\linewidth}
    \includegraphics[width=\linewidth]{Figs/qual-gt-Bark.png}
    \caption{Ground-truth for label `Bark'}
    \label{fig:qual_gt_bark}
\end{subfigure}
\centering
\begin{subfigure}[l]{\linewidth}
    \includegraphics[width=\linewidth]{Figs/qual-gt-Cough.png}
    \caption{Ground-truth for label `Cough'}
    \label{fig:qual_gt_cough}
\end{subfigure}
\centering
\begin{subfigure}[l]{\linewidth}
    \includegraphics[width=\linewidth]{Figs/qual-output-single.png}
    \caption{Output of the single-target extraction of `Bark'}
    \label{fig:qual_output_single}
\end{subfigure}
\centering
\begin{subfigure}[l]{\linewidth}
    \includegraphics[width=\linewidth]{Figs/qual-output-multi.png}
    \caption{Output of the multi-target extraction of `Bark' and `Cough'}
    \label{fig:qual_output_multi}
\end{subfigure}
\caption{{Visualization of time-domain waveforms of single-target and multi-target extraction (x-axis represents time). }}
\label{fig:qual}
\vskip -0.2in
\end{figure}


\vskip 0.05in\noindent{\bf Results.} We separately train the models for single-target and multi-target extraction tasks, and evaluate them on our testset. For multi-target evaluation, we train our model as well as baselines to make predictions with multi-hot query vectors, as opposed to one-hot queries used in the single-target evaluation. { During the multi-target training, 1-3 foreground sounds are randomly selected as target sounds. This training method using arbitrary number of target sources helps the model learn multi-target embeddings. The same model configurations are used for both the single-target and multi-target experiments.  The Conv-TasNet and ReSepfromer baselines are also trained in the same way for the multi-target extraction task.


We also evaluate the real-time factors (RTFs) of the models on an Intel Core i5 CPU using a single thread. RTF is computed by measuring the runtime consumed by the models to process a 416 sample audio chunk (9.43 ms at 44.1 kHz), and dividing that with the chunk duration, 9.43 ms. {For the RTF measurement, we include the padding for dilated convolution layers in our model's DCC encoder and Conv-TasNet's Temporal Convolution Network (TCN) blocks, accounting for the entire receptive field. In case of the ReSepformer, using a single chunk for RTF measurement excludes the overhead caused by causal attention masking in its inter-attention blocks. Consequently, the RTF value reported for ReSepformer is a lower bound of what is practically achievable.}}

{
Table~\ref{table:single_target} compares our models with different configurations with the baselines in terms of both the efficiency and the performance. We show that our approach  results in 2.2-3.3 dB SI-SNRi improvement compared with the  baselines, while being 1.5-2x more computationally efficient with 1.2-4x fewer parameters. Table~\ref{table:multi_target}  compares the performance of our models with the baselines for the multiple  target extraction task. It shows  that our method outperforms the baselines by 1.2-1.4 dB for the 2-target case, and 1-1.2 dB for the 3-target case. As with prior work~\cite{2020arXiv200605712O}, the SI-SNR improvements are  lower in the 3-target selection task since there is greater similarity between the input mixture and the target signal, compared to the single-target  case, resulting in a larger input SI-SNR.}

{
In Fig.~\ref{fig:qual}, we qualitatively show an example single-target extraction and multi-target extraction from a 4-class input mixture, using our multi-target extraction model. Fig.~\ref{fig:qual_mixture} shows the input mixture waveform, and Figs.~\ref{fig:qual_gt_bark} and \ref{fig:qual_gt_cough} show the isolated ground-truth sounds. We provide the input mixture to our multi-target model, with a single target query followed by a two target query. Figs.~\ref{fig:qual_output_single} and \ref{fig:qual_output_multi} are the output waveforms obtained when the single target and the two targets are queried, respectively. The waveforms show that the model successfully recognizes the queried events and extracts the relevant sounds. It can also be observed that our model preserves the original amplitudes of the sounds in the input mixture well.
}



The use of higher dimensionality such as 512, in DCC layers is shown to perform better \cite{luo2019conv}. Using the 512 dimensions in the transformer decoder, however, can make it computationally intensive. So, it is  better to run the DCC encoder at higher number of dimension than that of the transformer decoder. Since we use projection layers to convert between the encoder and decoder dimensions,  to mitigate the effects the projection layers have on gradients, we include a skip connection in the decoder. Our ablation study shows that the skip connection improves the SI-SNRi from 9.06 to 9.43.


\begin{table}[t!]
\caption{SI-SNRi  comparison in the multi-target extraction task.}
\vspace{-.8em}
\centering

\begin{tabular}{l  c c c c}
\toprule
 & \multicolumn{3}{c}{\# selected classes} \\
Model                   & 1 & 2  & 3 & Mean \\
\midrule
Conv-TasNet              & 7.20 & 3.63 & 0.19 & 3.67 \\
ReSepformer              & 7.42 & 3.56 & 0.33 & 3.77 \\
Ours ($E = 256;~D = 128$)  & 9.06 & 4.78 & 1.51 & 5.11 \\
Ours ($E = 256;~D = 256$)  & 9.12 & 4.76 & 1.31 & 5.06 \\
Ours ($E = 512;~D = 128$)  & 9.39 & 4.92 & 1.39 & 5.23 \\
Ours ($E = 512;~D = 256$)  & 9.29 & 4.92 & 1.35 & 5.19 \\
\bottomrule
\end{tabular}
\vskip -0.15in
\label{table:multi_target}
\end{table}
