Real-time single speaker recognition I'm developing an application for PC that records microphone data in real-time, and plays it back after applying some effects. 

I'd like to apply these effects only when a single known speaker is talking. The environment might include other people's voice and indoor noise (for example, a classroom or a restaurant).

Since that single speaker is also holding the microphone, I tried to simply filter input data based on a (configurable) volume level. This method is working but not very well, since sometimes the environmental noise might go above the threshold, or the speaker's voice might go below it.

Is there a better way to recognize a single speaker? I've read about speaker identification using feature extraction and HMM/GMM, but I'm not sure whether it is an overkill for my case.

I apologize if this it too vague. I'm new to DSP so I'm not really sure what additional information is needed to answer this.
Thanks!



Edit:


Speakers usually speak one after another, but background chatter is possible.
In addition, although the speaker is known, I do need some kind of an initial setup procedure (to adjust the application to that speaker).
I was going to suggest the machine learning route but a quick search turned up OSS so you might not have to re-invent the wheel: Identify speakers with sndpeek

Do the speakers talk over each other (which complicates things) or is it mostly one after another?