deep learning for text and sequences figure training and validation loss on he jena temperatureforecast ing task with gru the new validation mae of ~ (before you start significantly overfittingtranslates to mean absolute error of @ after denormalization that' solid gain on the initial error of @cbut you probably still have bit of margin for improvement using recurrent dropout to fight overfitting it' evident from the training and validation curves that the model is overfittingthe training and validation losses start to diverge considerably after few epochs you're already familiar with classic technique for fighting this phenomenondropoutwhich randomly zeros out input units of layer in order to break happenstance correlations in the training data that the layer is exposed to but how to correctly apply dropout in recurrent networks isn' trivial question it has long been known that applying dropout before recurrent layer hinders learning rather than helping with regularization in yarin galas part of his phd thesis on bayesian deep learning, determined the proper way to use dropout with recurrent networkthe same dropout mask (the same pattern of dropped unitsshould be applied at every timestepinstead of dropout mask that varies randomly from timestep to timestep what' morein order to regularize the representations formed by the recurrent gates of layers such as gru and lstma temporally constant dropout mask should be applied to the inner recurrent activations of the layer ( recurrent dropout maskusing the same dropout mask at every timestep allows the network to properly propagate its learning error through timea temporally random dropout mask would disrupt this error signal and be harmful to the learning process yarin gal did his research using keras and helped build this mechanism directly into keras recurrent layers every recurrent layer in keras has two dropout-related argumentsdropouta float specifying the dropout rate for input units of the layer see yarin gal"uncertainty in deep learning (phd thesis),october yarin/blog_ html licensed to