1. A method of speech-based user authentication, comprising: at a device comprising one or more processors and memory: receiving a spoken utterance of a speaker; generating a phoneme-independent matrix based on the spoken utterance, wherein the phoneme-independent matrix comprises a plurality of phoneme-independent feature vectors each having been extracted from a respective frame sampled from the spoken utterance at a sampling frequency; decomposing the phoneme-independent matrix into multiple sets of vectors including at least a first set of vectors defining at least one speaker-specific recognition unit and a second set of vectors defining at least one content reference sequence; computing at least one speaker-specific distribution value based on at least the speaker-specific recognition unit; and authenticating an input speech signal based on the at least one speaker-specific distribution value.