Real-time single speaker recognition
I was going to suggest the machine learning route but a quick search turned up OSS so you might not have to re-invent the wheel: Identify speakers with sndpeek

Do the speakers talk over each other (which complicates things) or is it mostly one after another?