(CNN+)RNN-HMM hybrid for learning phonemes from a spectogram I am currently working on a speech recognition task, on applying deep learning onto the standard acoustic model (gmm-hmm). 

I've currently generated a spectrogram of my utterances, and using simple pattern recognition managed to receive a 40%WER on a yes/no dataset.. Not great but a start.  The CNN is being fed a context window of 40 frames, in which the center frame is being detected for, my question is whether the use of an RNN could benefit here? so that the RNN handles the context, and the CNN does "image analysis" on one frame spectrogram.  

and if so will it implementation wise cause some problems, When I was doing it with a CNN, was the context dependency resolved by doing pattern recognition on a bigger part of the spectrogram, depending of the context window size, but introducing RNN, CNN only has to do analysis on one frame (which I see can become to even get a proper results from) at the time, and can the one framed information be piped to the RNN until a certain context size has reached? and if so how?
Have you seen this... 

http://ieeexplore.ieee.org/document/7953168/


  ... We propose to use a recently developed deep learning model, recurrent convolutional neural network (RCNN), for speech processing, which inherits some merits of recurrent neural network (RNN) and convolutional neural network (CNN). The core module can be viewed as a convolutional layer embedded with an RNN, which enables the model to capture both temporal and frequency dependance in the spectrogram of the speech in an efficient way.


Sound pretty accurately to what you are searching for, or trying to implement.