What do we mean by Total Factor Space in Audio Processing?
I read the paper, let me try to summarize what I understand.

1) For speaker identification tasks, systems need to be robust to  "session variability". This means that from one spoken utterance to another, the way a speaker pronounces the same passphrase &amp; the environmental/background noise impacts adds a bit of variance to the captured audio signal. I.e. it is not exact. 

2) Historically, speaker-verification techniques based on GMM-UBM->MAP are not robust to channel distortion (i.e distortion during the recording) or phonetic variability (i.e. differences in pronunciation) or other "nuisance factors"



3) A single space named Total Factor Space is constructed to model speaker and channel variability jointly. Joint factor analysis (JFA) is one technique which decomposes a speaker's "mean supervector"  (which is a linear combination of the set of eigenvoices) into speaker component S and channel component C

So instead of doing separate modeling of speaker and channel spaces, all factors (i.e. both speaker and channel) are combined into one space. The JFA model from the slideshare below looks like: 



Have a deeper look at this slideshare. Good luck!