comparing 2 audio files in c#
I'm thinking the signals could be compared by 

$D = \mathcal{F}^{-1}(\mathcal{F}(A) - \mathcal{F}(B))$

Where A and B are the two audio files to be compared. 

I guess after that you could come up with some criteria for deciding that the voices are different. Perhaps when $|D|$ goes over some threshold.

You could also use Cross correlation

Using some other library to help with this would save time. But I haven't done this with audio files. 

AForge Looks like it has FFT although I haven't used it.