(CNN+)RNN-HMM hybrid for learning phonemes from a spectogram
Have you seen this... 

http://ieeexplore.ieee.org/document/7953168/


  ... We propose to use a recently developed deep learning model, recurrent convolutional neural network (RCNN), for speech processing, which inherits some merits of recurrent neural network (RNN) and convolutional neural network (CNN). The core module can be viewed as a convolutional layer embedded with an RNN, which enables the model to capture both temporal and frequency dependance in the spectrogram of the speech in an efficient way.


Sound pretty accurately to what you are searching for, or trying to implement.