When does the laugh track kick in? One of my weekend projects has brought me into the deep waters of signal processing. As with all my code projects that require some heavy-duty math, I'm more than happy to tinker my way to a solution despite a lack of theoretical grounding, but in this case I have none, and would love some advice on my problem, namely: I'm trying to figure out exactly when the live audience laughs during a TV show.

I spent quite a bit of time reading up on machine learning approaches for detecting laughter, but realized that's more to do with detecting individual laughter. Two hundred people laughing at once will have much different acoustical properties, and my intuition is that they should be distinguishable through much cruder techniques than a neural network. I may be completely wrong, though! Would appreciate thoughts on the matter.

Here's what I've tried so far: I chopped up a five minute excerpt from a recent episode of Saturday Night Live into two second clips. I then labeled these "laughs" or "no-laughs". Using Librosa's MFCC feature extractor, I then ran a K-Means clustering on the data, and got good results -- the two clusters mapped very neatly to my labels. But when I tried to iterate through the longer file the predictions didn't hold water.

What I'm going to try now: I'm going to be more precise about creating these laughter clips. Rather than do a blind split and sort, I'm going to manually extract them, so that no dialogue is polluting the signal. Then I'll split them into quarter second clips, calculate the MFCC's of these, and use them to train a SVM.

My questions at this point:


Is any of this making sense?
Can statistics help here? I've been scrolling around in Audacity's spectrogram view mode and I can see pretty clearly where laughs occur. In a log power spectrogram, speech has a very distinctive, "furrowed" appearance. In contrast, laughter covers a broad spectrum of frequency quite evenly, almost like a normal distribution. It's even possible to visually distinguish applause from laughter by the more limited set of frequencies represented in applause. That makes me think of standard deviations. I see there's something called the Kolmogorov–Smirnov test, might that be helpful here?

(You can see the laugh in the above image as a wall of orange hitting at 45% of the way in.)
The linear spectrogram seems to show that the laughter is more energetic in lower frequencies and fades out towards the higher frequencies -- does this mean it qualifies as pink noise? If so, could that be a foothold on the problem?



I apologize if I misused any jargon, I've been on Wikipedia quite a bit for this one and wouldn't be surprised if I got some jumbled.
Based on your observation, that spectrum of the signal is sufficiently distinguishable, you can use this as a feature to classify laughter from speech. 

There are many ways you can look at the problem. 

Approach #1

In once case, you can just look at vector of the MFCC. and apply this to any classifier. Since you have many co-efficient in frequency domain, you might want to look at Cascade Classifiers structure with boosting algorithms such as Adaboost based on this, you can compare between speech class vs. laugh class.

Approach #2

You realise that you speech is essentially a time varying signal. So one of the effective ways to do it is to look at time variation of the signal itself. For this, you can split signals in batches of samples, and look at the spectrum for that time. Now, you may realise that laugh might have more repetitive pattern for a stipulated duration where as speech inherently posses more information and hence the spectrum variation would be rather bigger. You can apply this to HMM type of model to see if you consistently remain within the same state for some frequency spectrum or you continuously keep changing. Here, even if occasionally the spectrum of speech resembles to that of laughter it will be more time changing. 

Approach #3

Force to apply LPC/CELP type of coding on the signal and observe the residue. CELP Coding makes a very accurate model of speech production. 

From the reference here: THEORY OF CELP CODING 


  The redundancies in the speech signal are almost removed after the
  short term prediction and long term prediction of the speech signal
  and the residual has very little correlation left in it. Then an
  excitation is searched which synthesizes the speech and the codebook
  index and gain are searched from the fixed codebook. The optimum
  codebook index selection criterion is based on MMSE between the
  locally synthesized speech and the original speech signal.


To put it simply, after all the speech which is predicted from analyser is removed - what is left is the residue which is transmitted to recreate exact waveform. 

How does that help with your problem? Basically, if you apply CELP coding, speech in the signal is mostly removed, what remains is residue. In case of laughter a majority of the signal might be retained because CELP will fail to predict such a signal with vocal tract modelling, where as individual speech will have very little residue. You can also analyse this residue back in frequency domain, to see if it is laughter or speech.