1. In a system where an annotation guide is used to label utterances in speech data with a call type, a method for monitoring labelers of the speech data, the method comprising: presenting via a processor a test utterance to a labeler; receiving input from the labeler that selects a particular call type from a list of call types; determining via the processor if the labeler labeled the test utterance correctly; and based on the determining step, performing at least one of: revising the annotation guide, retraining the labeler or altering the test utterance.
1. In a system where an annotation guide is used to label utterances in speech data with a call type, a method for monitoring labelers of the speech data, the method comprising: presenting via a processor a test utterance to a labeler; receiving input from the labeler that selects a particular call type from a list of call types; determining via the processor if the labeler labeled the test utterance correctly; and based on the determining step, performing at least one of: revising the annotation guide, retraining the labeler or altering the test utterance. 2. A method as defined in claim 1 , wherein presenting a test utterance to a labeler further comprises selecting the test utterance from a group of labeled utterances.