How do MS-TDNNs work? Multi-State Time Delay Neural Networks (MS-TDNNs) were introduced in


  Haffner, Patrick and Waibel, Alex: Multi-state time delay networks for continuous speech recognition. In Advances in neural information processing systems, 1992.


They are an extension of TDNNs. TDNNs are convolutional neural networks for automatic speech recognition (ASR), where the convolution happens over the time.

The aim of MS-TDNNs seems to be to get rid of the hybrid approach in ASR, where you need dynamic programming / HMMs to chunk the audio stream and then neural networks to recognize the phonemes. Somehow MS-TDNNs seem to do the segmentation as well.

I don't understand how. Could somebody please explain it to me?

(Related side questions: Are MS-TDNNs recurrent networks? Where exactly does the name "multi-state" come from?)
So I think you understood that a MS-TDNN has two parts: a conventional TDNN that calculates state probabilities for each frame and kind of a perceptron on top of it linking the states of several frames to a word. The later part is doing the segmentation. Its connections aren't trained, but just used for running BP and train the TDNN.

The segmentation is obtained by some other algorithm (I don't remember exactly, but something like dynamic time warping). In normal DNN-HMM hybrid systems the DNNs are trained separately to predict the correct state for each frame. The error function is calculated on frame level!

This is not optimal as we do not care so much about each state being correct, but the final text. WER is calculated on world level. MS-TDNNs try to solve this by having this extra layer that allows passing a word level based error to BP. This is very similar to todays methods called sequence training.

Neither TDNN nor MS-TDNN are recurrent networks.

If you are interested in RNNs and ASR systems that do not require HMMs I would recommend you to look at CTC objective that does indeed learn the alignment automatically.