1. A system for performing a speech verification of an utterance, comprising: a search module configured to generate an N-best list of recognition candidates corresponding to said utterance; a pseudo filler score calculator coupled to said search module for determining a pseudo filler score that is based upon recognition scores corresponding to said recognition candidates; a confidence measure calculator for utilizing said pseudo filler score, a background score of a worst recognition candidate, and a best recognition score of a best recognition candidate to determine a confidence measure; and a comparator configured to compare said confidence measure to a threshold value for determining whether to verify said best recognition candidate.