Recognition of numbers from audio
Not aware of research per se but I did this in C# before. The concept you need to know about to recognize only certain patterns in speech is "Grammar". 

Grammars in terms of speech or text recognition consist of multiple rules and word tokens that can be identified by your speech recognizer. 

As further inspiration for your idea maybe look at SpeechRecognizer and the Grammar class in the VB.net Framework.

ie:
https://docs.microsoft.com/en-us/previous-versions/office/developer/speech-technologies/hh361633(v=office.14)?redirectedfrom=MSDN