Audio Analysis : Segment audio based on speaker recognition
Initially I did using silence detection but later moved to pyAudioAnalsis which is better.   

Check "Speaker Diarization" section in Segmentation in pyAudioAnalysis