Incremental Learning Algorithm for Sound Event Detection
Eunjeong Koh Fatemeh Saki Yinyi Guo Cheng-Yu Hung Erik Visser
cs.LG cs.SD eess.AS




𝐱
ŁL










Eunjeong KohWork performed as an intern at Qualcomm Technologies
	University of California, San Diego

	Department of Music

	La Jolla, CA, USA

	
 Fatemeh Saki, Yinyi Guo, Cheng-Yu Hung, Erik Visser
	Qualcomm Technologies, Inc.,

	San Diego, CA, USA

	

	








Incremental Learning Algorithm for Sound Event Detection
    Matthew S. Fox
    December 30, 2023
========================================================






This paper presents a new learning strategy for the Sound Event Detection (SED) system to tackle the issues of i) knowledge migration from a pre-trained model to a new target model and ii) learning new sound events without forgetting the previously learned ones without re-training from scratch. In order to migrate the previously learned knowledge from the source model to the target one, a neural adapter is employed on the top of the source model. The source model and the target model are merged via this neural adapter layer. The neural adapter layer facilitates the target model to learn new sound events with minimal training data and maintaining the performance of the previously learned sound events similar to the source model. Our extensive analysis on the DCASE16 and US-SED dataset reveals the effectiveness of the proposed method in transferring knowledge between source and target models without introducing any performance degradation on the previously learned sound events while obtaining a competitive detection performance on the newly learned sound events.




Incremental learning, sound event detection, transfer learning





§ INTRODUCTION



Sound Event Detection (SED) is a rapidly growing research area that aims to analyze and recognize a variety of sound events in a continuous audio signal. Neural Networks based methods such as Convolutional Neural Networks (CNNs) have recently been used for SED systems to advance the performance of these systems <cit.>. In the Detection and Classification of Acoustic Scenes and Events (DCASE) Task 4 <cit.>, the state-of-the-art SED systems have been tested using real data that is either weakly labeled or unlabeled and simulated strongly labeled data with the onset and offset times of sound events. 





Although SED problem has been attracting many researchers, a vast majority of the state-of-the-art systems are focused on advancing the performance of the SED systems by utilizing weakly labeled data <cit.>. To our knowledge, one of the important and non-investigated challenges of the current SED models is their closed-set nature, where a fixed and limited number of known classes are used during the training. It is difficult to collect exhaustive training samples or to properly annotate all the training data to train the classifiers. Hence in the closed-set classifiers, only a limited number of classes are considered for training, with the assumption that during test time, the test data is drawn from the same set of classes as the training data. However, the SED systems in nature are open-set problems, in other words, the test data could include samples associated
with unknown sound events as well. Therefore, it is always desired to have a flexible model that can learn new classes, once new training data including new sound events becomes available. Then again, it is required to still remember the previously learned classes after adopting the new classes, and learning capability is referred to as continuous learning or incremental learning. 

One of the main challenges associated with these types of continuous learning algorithms is catastrophic forgetting <cit.>. That is if the information about the previously learned categories is unavailable when a new task is added, it overwrites the previously learned information. Hence, it leads to the performance degradation of past tasks. The ability of continuous/incremental learning over time represents a long-standing challenge for Machine Learning and Neural Networks <cit.>. Recently, in the areas of computer vision and natural language processing <cit.>, Transfer Learning (TL) has shown great potential to 1) identify the transferable knowledge by accommodating new knowledge and 2) retain previously learned information. Some recent works have explored TL for audio applications <cit.>, which focus on knowledge transfer between databases with various qualities, mismatch downstream tasks, and domains. However, it remains to be seen how a flexible TL model for SED task to audio knowledge transfer can be done.














In this paper, we present an incremental learning algorithm for SED applications effectively transferring knowledge from a source model to a target model. Our method updates the target model when new sound events are available without any catastrophic forgetting. Motivated by the recent TL advances in natural language processing <cit.>, we utilize a neural adapter to bridge the gap between the source model and the target model. We combine new neurons for transferring parameters from the source model and implement a neural adapter to lessen the gap between the source and the target data distribution. For testing the performance of the neural adapter, we test the basic simple TL approach, then we show the impact of our proposed neural adapter for the SED task, while testing several TL options on just one category. The results show that our method provides an effective knowledge transfer mechanism between source and target domains without any additional training examples and any performance degradation of the previously learned tasks in the source domain. Our learning algorithm helps to transmit the predictions from the source model into that of the target model.




The rest of the paper is organized as follows. In section 2, we cover the proposed SED incremental learning algorithm, followed by the experimental results in section 3. The conclusion and the discussion are provided in section 4.





§ SED INCREMENTAL LEARNING ALGORITHM





 §.§ SED Problem Formulation


Given an input audio file that includes several acoustic scenes, a standard SED predicts the corresponding labels that indicate the annotation of all the sound events in the scenes. The time-series audio input is represented by audio embedding vectors and the event label file includes the information of event specification, such as the sound event onset/offset time and sound event label. 




 §.§ Transfer Learning (TL) Workflow


Our TL mechanism consists of several steps (see Figure <ref>); (i) a pre-trained source model, M_S, for a certain number of categories, in the source domain, D_S; (ii) a target model, M_T, that utilizes the source model M_S parameters as a starting point to learn the target domain, D_T. In addition to the typical TL mechanism, which is (i) and (ii), we add a neural adapter (iii) a connection system between the source model M_S and the target model M_T for effectively transferring knowledge from the source domain D_S to the target domain D_T. This connection will alleviate the effect of information discrepancy and prevent any catastrophic forgetting on the previously learned information. Finally, we (iv) jointly train the target model M_T and the connection system together to effectively learn the target domain D_T information. Note that the parameters of the source model M_S are not updated during this target model M_T training process.







 §.§ Model Architecture





  §.§.§ SED Source Model


Figure <ref> describes the process and the structure of our SED incremental learning methodology. We revise the Convolutional Neural Network (CNN) proposed by Salamon and Bello <cit.>, which includes three 2D convolution layers, a 2D max-pooling operation, and batch normalization layers. Each layer processes 64 convolutional filters. The input to the network is a Mel spectrogram of size 128x128 that is extracted from a one-second audio file. ReLU activation functions are applied to the convolutional layers 
to reduce the backpropagation errors and accelerate the learning process <cit.>. Sigmoid function 
are used as the output activation function with N classes. Adam optimizer <cit.> and binary cross-entropy loss function are used. The stopping criterion is set as 500 epochs with an early-stopping rule, if there is no improvement to the F1 score during last 100 learning epochs <cit.>. The final model has 720k parameters. This model is implemented in Keras <cit.>.




  §.§.§ Incremental Learning using Neural Adapter




In this work, the M_S is trained for N sound events using the aforementioned CNN model. The goal is to create a M_T for N+1 sound events without training from scratch. Note that the N sound events are common for both M_S and M_T. To this end, the M_T has the same CNN structure as the M_S and the trained parameters of the previously learned N sound events are utilized from the M_S as an initial training point for the M_T. To account for the new category in the M_T, we modify its output layer with N+1 sigmoid activation. It is well known that learning a new task via such a simple transfer learning paradigm usually results in forgetting the previously learned classes while adding new classes. To avoid this knowledge-lost problem, we adopt the TL mechanism proposed by Chen and Moschitti <cit.>. In this method, to effectively transfer the knowledge between the M_S and the M_T, a neural adapter is utilized to bridge the two models and jointly trained with the M_T. More specifically, a neural adapter consists of two fully-connected dense layers over the last layer of M_S is used to connect the M_S to the M_T. This process is called the element-wise summation which integrates the outputs from the source and target domain and finally processes N+1 categories. The parameters of the neural adapter and the M_T are learned simultaneously while the M_S parameters are fixed.



§ EXPERIMENTAL RESULTS





 §.§ Datasets


We start our implementation with building M_S using only N sound events for D_S, out of the N+1 events, leaving one sound event out to be learned incrementally later with the M_T. We train the M_S using the D_S until the optimal parameters are achieved. These parameters and results will be saved and re-utilized for D_T. We evaluate our algorithm over three datasets; the DCASE 2016 challenge Task 2 (DCASE16) <cit.>, the UrbanSound-SED (US-SED), and UrbanSound-8K (US-8K) <cit.> dataset.  




The DCASE16 dataset includes eleven different sound events for the SED challenges[<http://www.cs.tut.fi/sgn/arg/dcase2016/task-sound-event-detection-in-synthetic-audio>], clearing throat, coughing, door knock, door slam, drawer, human laughter, keyboard, keys (put on the table), page turning, phone ringing, and speech. In this work, we use four sound events; “door knock", “door slam", “keyboard", and “phone ringing". To generate soundscapes from these sound files, we use the Scaper open-source library <cit.> for the synthesis and augmentation [We revise this open-source implementation: 

<https://github.com/justinsalamon/scaper_waspaa2017>].  We create 800 soundscapes for M_S training data and 200 soundscapes for each test/validation data. The duration of each soundscape audio file is ten seconds. It is worth noting that the soundscapes are generated in such that each event appears at least once and a maximum of two times in every soundscape file. This DCASE16 dataset denotes a clean and well-labeled dataset for our experimental setting.




The US-SED dataset <cit.> includes ten different sound classes; air conditioner, car horn, children playing, dog bark, drilling, engine idling, gun shot, jackhammer, siren, and street music. In this work, we use the pre-generated UrbanSound soundscape audio files from the Scaper study <cit.>. It has 10,000 soundscape files with a duration of ten seconds. Each of the soundscape files has a minimum number of sound events as zero, and the maximum number of nine. Thus, some of the soundscapes might be empty of the sound events of interest. This situation can be interpreted as a more realistic SED framework compared to the aforementioned DCASE16, where each of its soundscape files contains all the sound events. For our experiment, we considered five sound events; “car horn", “dog bark", “gun shot", “siren", and “street music". 4,995 files are used for M_S training, and 1,665 files for each of the test and verification data. 




  



 §.§ Performance of the Incremental Learning


We evaluate our algorithm in three different settings; i) evaluating the M_S trained on D_S with N sound events, ii) evaluating the M_T that is built via a simple TL on the D_T with N+1 sound events, and iii) evaluating the model that is built utilizing the neural adapter, where the M_T is merged with the M_S through the neural adapter and trained on D_T. The results are reported in terms of F1-score (see Table <ref> and <ref>). We perform experiments per every class assuming it is newly introduced in the D_T. Note that N indicates three sound event classes for DCASE16 dataset and N indicates four sound event classes for the US-SED and US-8K dataset.





Table <ref> and Table <ref> present the evaluation results for the DCASE16 and US-SED, respectively. These tables include three sections; M_S, Simple TL and Neural Adapter TL. The M_S section shows the F1-score of M_S models for different D_S. Section Simple TL illustrates the F1-score of the M_T that are built via a simple transfer learning approach. Section Neural Adapter TL reports the F1-score of the proposed approach. D_S columns of the Simple TL and the Neural Adapter TL indicate the F1-score of the new models on the previously learned classes after the incremental learning process. The New columns present the F1-score of the newly learned class, and finally the All columns report the overall F1-score on the N+1 sound events. Rows in the table depict different test scenarios assuming that unseen label in D_S is newly introduced in D_T for incremental learning. 




∙ DCASE16: By paying attention to Table <ref> and the D_S column in the M_S section versus the D_S column in the Simple TL section, it can be seen that learning sound classes via the simple TL approach, without the neural adapter, results in performance degradation in the previously learned sound events. This drop in performance indicates a catastrophic forgetting in the D_T on the original three categories in D_S. On the other hand, the consistent F1-scores between the D_S column in the M_S section and the D_S column in the Neural Adapter TL section is an illustration that the neural adapter can properly maintain the knowledge learned from D_S while learning the new class in the D_T. This result proves that the neural adapter manages to mitigate the knowledge forgetting and enabling the model to update to the new domain. 







∙ US-SED: The top table of Table <ref> illustrates the results on US-SED dataset. Similar to DCASE16, we can see the neural adapter can effectively bridge the knowledge between the M_S and the M_T models while learning the new sound events. However, by comparing the D_S column in the M_S section to the D_S column in the Simple TL section, we can see the simple TL achieves some improvement over the M_S on the previously learned sound events. Unexpectedly, these results are even slightly better than the neural adapter TL approach (see the D_S column in the Neural Adapter TL section). This result does not match the results obtained for the DCASE16 dataset. The reason for this could be the amount of presented noise during the training of the models. As it is mentioned earlier, in the US-SED, some of the used soundscapes in the training data are empty of the sound events of interest. Hence, these files are interpreted as noisy samples by the network. On the other hand, all the sound files used from the DCASE16 at least contain one of the desired sound events. To assess this argument, we modify the original UrbanSound8K dataset <cit.> to mimic the settings from the DCASE16 in creating a more clean dataset[For our experimental setting, US-8K soundscapes are generated based on the UrbanSound8K dataset which can be found on: 
 <https://urbansounddataset.weebly.com/urbansound8k.html>.] and this is the US-8K for our next dataset. In US-8K, each event appears at least once and a maximum of two times in every soundscape file which is the same setting as DCASE16 soundscape generation. 





∙ US-8K: This US-8K dataset has the same five sound events as the US-SED. Similar to DCASE16, each sound event appears in each soundscape file at least once. The evaluation results on this dataset is provided in the bottom table of Table <ref>. By looking at the D_S column in the M_S section and the D_S column in the Neural Adapter TL section, it can be seen that similar to the DCASE16 dataset, the neural adapter method consistently maintains the performance on the previously learned sound events after learning new sound events. Also transferring the knowledge from the M_S via the neural adapter is more effective compared to the simple TL approach. It is important to ensure that this improvement in the performance is not specific to any target event category, and it is common across different experiments denoted in various rows of the tables. 




Summarizing the experimental results from the three datasets, it can be seen that simple TL method without the neural adapter has confronted the degradation of the performance with losing the previously learned knowledge from the source model training. There are also up and down in the performance from specific sound events or dataset in the simple TL method. On the other hand, in the case of the neural adapter approach, it is possible to see the inclination in which the learned knowledge is maintained consistently, and the performance sustains in a balanced manner. Therefore, we show performance consistency on the previously learned sound events through neural adapter while obtaining decent detection performance on the newly learned sound events well.





 §.§ The Feasibility of Neural Adapter for Incremental Learning




In this section, we study the contribution of each model's outputs separately on the overall performance. In Figure <ref>, we separate three individual outputs in the neural adapter approach; (1) the optimal output of the source model with a neural adapter (A in Figure <ref>), (2) the output of the target model (B in Figure <ref>) and (3) the final output of our proposed TL model (C in Figure <ref>). For this study, the target domain is used in three cases. The results of the analysis are provided in Table <ref>. 



It can be seen that when using DCASE16 dataset, the trained target model via the neural adapter reaches an optimal point to be able to detect all the N+1(=4) sound events without the need to have the source model (see B&C of the DCASE16 column). Therefore, only the target model could be stored and used as a starting point for learning new categories without the need to store the source model and the neural adapter. This result can provide a low footprint continuous learning framework for further model expansion. 




In contrast, in the case of using the US-SED dataset, the target model and the source model remain complementary to each other for achieving an acceptable final outcome (see B&C of the US-SED column). It appears that the target model cannot maintain the transferred knowledge from the source model while learning the new sound event. Hence, in order to continuously learn new sound events, we always need to keep the whole TL structure presented in Figure <ref>. This is not feasible for continuous learning in applications that are operated on resource limited platforms, for example, wearable devices. In the case of using the US-8K dataset, it has similar aspect of DCASE16, but it is hard to see a difference as large as DCASE16. 







§ CONCLUSION AND DISCUSSION


We present an incremental learning algorithm utilizing a TL paradigm for SED application. We use a neural adapter to effectively bridge the gap between the previously learned information in the source model and a target model for learning new sound events. Our extensive analysis shows that utilizing such a mechanism improves the performance of recognizing both known/unknown sound events without forgetting the previously learned knowledge. Thus, our proposed model suits well the scalable and incremental SED applications. 




This approach can also be used as a low footprint framework for continuous learning in applications that involve less noisy and well annotated data. However, for the more realistic applications, such as acoustic scene classification systems that involve more noisy data, both the target model and the source model might need to remain connected to achieve the desirable performance. Addressing such a challenge remains the focus of our future work. 















IEEEbib






