Converting a speech recognition model from CNNs to GRUs
I think your network's design is fine but need change the way you feed training data.
In essence you try convert a classic CNN (having &quot;image&quot; data, but spectrograms) to a sequential time-domain type of network (recurrent one, having GRU or LSTM at the heart).
To be able to do that you have to reconsider the way you feed the input data:

Input data should not be the &quot;image&quot; snapshots of spectra (as expression of audio spectra in a time-window) but instead have to feed evolving sequences like text ones (the .wav audio samples in sequential way).

If still would like to feed with spectrograms and not raw sequential wave samples, try feed the spectrograms by cloning each of them into time evolving variants having offset by t, t+1, t+2, t+N variants of same audio snapshot (recompute FFT as many times) thus recurrent GRU unit may have chance to capture sequence information out if it.


Also:

BatchNorm &amp; Relu was added, a good way to improve generalization and overfit, at the price that might requires more data to see benefits.
Pooling &amp; Flatten summarisation was replaced by GRU recurrent unit, but these units learns only when capturing from time evolving data.
Very Conv2d in front of GRUs helps generalization but Conv2d x times x GRU also means lots of combinatoric space opened up, implict might require more input data or prolonged training.

Anyway is a fun experiment worth trying !