deep learning for text and sequences other types of recurrent layers perform much better let' look at some moreadvanced layers understanding the lstm and gru layers simplernn isn' the only recurrent layer available in keras there are two otherslstm and gru in practiceyou'll always use one of thesebecause simplernn is generally too simplistic to be of real use simplernn has major issuealthough it should theoretically be able to retain at time information about inputs seen many timesteps beforein practicesuch long-term dependencies are impossible to learn this is due to the vanishing gradient probleman effect that is similar to what is observed with non-recurrent networks (feedforward networksthat are many layers deepas you keep adding layers to networkthe network eventually becomes untrainable the theoretical reasons for this effect were studied by hochreiterschmidhuberand bengio in the early the lstm and gru layers are designed to solve this problem let' consider the lstm layer the underlying long short-term memory (lstmalgorithm was developed by hochreiter and schmidhuber in ; it was the culmination of their research on the vanishing gradient problem this layer is variant of the simplernn layer you already know aboutit adds way to carry information across many timesteps imagine conveyor belt running parallel to the sequence you're processing information from the sequence can jump onto the conveyor belt at any pointbe transported to later timestepand jump offintactwhen you need it this is essentially what lstm doesit saves information for laterthus preventing older signals from gradually vanishing during processing to understand this in detaillet' start from the simplernn cell (see figure because you'll have lot of weight matricesindex the and matrices in the cell with the letter (wo and uofor output output - output output_t activationwo*input_t uo*state_t bostate input - figure output + state + input input + the starting point of an lstm layera simplernn seefor exampleyoshua bengiopatrice simardand paolo frasconi"learning long-term dependencies with gradient descent is difficult,ieee transactions on neural networks no ( sepp hochreiter and jurgen schmidhuber"long short-term memory,neural computation no ( licensed to