RNN as posterior probability estimation in speech recognition with HTK I'm new to speech recognition and deep learning and in a learning phase.

I'm trying to follow this paper to learn how to use RNN as posterior probability estimation in an HTK environment. The paper proposes RNN-HMM hybrid system, so for the HMM part I need to use HTK platform.

The problem is that I couldn't even start from anywhere. I have a sample code which uses HMM to recognize digits, but I'm unable to solve at which part should I insert RNN to the code.

If there are any ideas, I would be glad.

The system in the paper is shown as follows:


I have codes in python environment and HMM is applied using HTK.
After converting data to MFCC format, I should use RNN, but after using RNN, which steps of HMM should I apply to generate RNN-HMM acoustic system.
It is very huge effort to build RNN-HMM system with HTK. It is easier to use more flexible toolkits like Kaldi, they have RNN-HMM already implemented.

HTK in latest version 3.5 beta already has support for DNN (not RNN but still a neural network), you can grep for HYBRIDHS in sources. For details on DNN-HMM it is better to read HTKBook 3.5 section 3.12 DNN-HMM Systems and Chapter 14 Artificial Neural Networks

You need to modify all places in HModel.c file to introduce your own type of the model. It is not a separate step you could introduce, you have to modify complex interaction between acoustic model and HMM search code.

It is also not straightforward to embed Python code into HTK C code, you might dump RNN scores in a file and read them from C, otherwise, you will have to reimplement your RNN code in C.