Speech recognition library for detecting phrase in file
The detection of a particular phrase in records sounded by different people may necessitate a complex speech recognition operation. Even if you train a good acoustic model, there is no guarantee that you will obtain a good recognition performance because you would also need some textual content which will constitute the language model. 

Google speech recognition performs quite good and you can use its API instead of developing a system from scratch:
https://stackoverflow.com/questions/8830203/is-there-an-api-for-googles-speech-recognition-technology

You can send segments of voice data to the API and get the recognition results.