How do I go about detecting whistles, pops and other sounds in live audio input? I've read many questions on SO, and frankly, each one of them is not describing any particular way to go about it. Some say "do FFT" and some say "zero crossing" etc. But I've only gone as far as understanding that the digital audio input consists of an array of amplitudes for a particular frequency and well.. Don't really know much beyond it.

Now I do know Nyquist theorem, frequency, amplitude, Fourier series etc, and that's from 2-3 years ago when I did that in my university program in some semester. But back then we really weren't taught the real world usage of Fourier much, and I didn't bother to dig up more on the subject beyond learning just enough to pass the subject. But now I'm gonna have to use all that stuff.

Here's a snapshot of the sounds I'm trying to detect:



Clearly the sounds have unique graphs. I just want to understand how to extract their particular unique characteristics for their unique spiky lines in the graph. Like what amplitude, frequency, etc. And for how much time - although that is trivial I guess.

I want a simple step by step, un-vague instruction list - I can google the terminology I won't understand.

Like perhaps this? --


Get input audio data
Plot spectogram
Get spectogram graph for the sound you want to detect in a noiseless environment
Study that graph - draw unique characteristics of that sound
Make some kind of function that can detect those particular characteristics in live audio feed, using the characteristics of the sound found in (4)
If a match is found, well - the job's done.
Polish the algorithm to weed out false negatives.


Where I am confused is - how do I go about 3,4,5. Particularly 3 and 4.
Because I'm making a software where in the user would be able to record any sound as a gesture input to be used later, I would want my system to be able to record any sound and match it against live audio feed to detect that sound gesture. Like say, coughing, or finger snapping, or whistling, or clapping. Or saying consonants - $A$ or $B$ or $X$ or $Z$. Just about any sound.

I was thinking of making the user record the sound they want to store as a gesture in a quite environment. And the user would make the sound only in between the quiet time padding; 3 secs in the beginning and in the end of the recording.

So say, during the first 3 secs, my system would establish that the current input is the normal quiet background sound. And then a sudden change in the graph would be the sound input starting. And when that stops, the recording would go on for another 3 secs, the trailing quiet time padding. This would be done manually by the user. It would then automatically store the characteristics of only that part during which the sudden change in the graph lasted - somewhere between the padding times.

And thus the characteristics of that part would be saved as that sound's gesture data, which would be used to detect that particular sound in live audio feed later on.

Problem is, that I'm thinking all this in plain English. I need to think in maths and physics, to be able to implement it efficiently in my code. I'm just so damn clueless about what to write and where to write it in my code - even with so many libraries and questions on SO at my disposal.

And sorry if this was long.
While I agree with one of the comments that using speech recognition techniques might be a good start, these sounds are different and I'm not aware of anyone having done any research into categorizing them (the paper cited by Nathan appears only to distinguish between speech and noise), so unless someone presents something to contradict that, you are going to have to invent your own technique, and that's going to take a lot of learning and a lot of work. The best I can do is get you started.

First off, don't expect anyone to be able to produce some magic formula. There's no equation to get from sound to what the sound is. Humans and computers must learn by incorporating data to make educated guesses about what sounds are. The reason people say in their answers "use FFT" or "use zero crossing" is because these are some of the basic DSP building blocks used in speech recognition and related algorithms. But the FFT and zero crossing rate are usually just first steps in creating a set of parameters describing the sound. These parameters are then analyzed statistically (not through some magic function) to determine which category they most likely belong to. Note that I said "most likely": even the best speech detection (and the human brain!) has a high error rate at this stage and further processing using knowledge of the language and grammar is used to correct these mistakes.

So, some parameters you could look for include:


zero-crossing rate
frequency centroid
envelope (this is actually a set of parameters, including, eg attack time)
spectral envelope
even/odd harmonic balance
voiciness
fundamental pitch


Once you have a set of parameters that you think will allow you to distinguish your sounds, you will need to use some statistical method to categorize them. Hidden Markov Model is often used in speech. You can also look into logistic regression, K-means, and I'm sure there are other choices, but I think HMM is tried and true.