Algorithm to detect if a spoken phrase matches a previously spoken phrase I would like to create a very simple system that allows a user to train (once or multiple times) a spoken phrase (from 1 word to a whole sentence.)

Then the (same) user would speak the phrase back, and the system would score how close to the original the phrase was matched. If certain words were missed or incorrect, the score would be deducted.

I've seen several articles about audio fingerprinting (ie Shazaam) but these implementations are an overkill. I have a known sound pattern that I'm matching against, not search against a database of audio.

Here's what I've tried (using the R libraries DTW and tuneR):


recorded the phrase "Known as the father of the constitution, he was the first to arrive in Philadelphia for the Constitutional Convention." three times, however, the 3rd time instead of saying "Philadelphia" I said "New York".
took the MFCC of all three WAV files
took the DTW three times: Phili1 vs Phili2, Phili1 vs. NY, Phili2 versus NY
Plotted the alignment, question is...what do I do with this data, what metric should I use You can see in the plot that the first alignment is most linear (same phrase repeated exactly)


Phili1 vs Phili2

Phili1 vs. NY

Phili2 vs. NY
MFCC is an established way of handling speech recognition. You will find plenty of libraries and tools (e.g. yaafe, SoX) that can extract them for you. Implementing them yourself is also quite easy. Yaafe provides plethora of other audio features you might like to play with.

Mind that MFCC are not very robust against noise, so you might like to apply some noise reduction beforehand if required. Some people advocate raising the log-mel-amplitudes to a power 2 or 3 before applying discreet cosine transform.

Some libraries, like scikit learn, next to all machine learning algorithms provide also DTW. On your place I would just try MFCC and see how it goes from there.