Understanding Layers in Recurrent Neural Networks for NLP In convolution neural networks, we have a concept that inner layers learn fine features like lines and edges, while outer layers learn more complex shapes.

Do we have any such understanding for layers in RNNs (like LSTMs), something like inner layers understand grammar while outer layers understand more complete meanings of sentences assuming that we are using the LSTM for some natural language task like text summarization?
Its not like it just understands grammar.

In LSTMs the network tries to preserve the hidden states over time. By doing this they try to learn long-term dependencies in the language and relationships between words at variable distances.

LSTM does this by using its three famous gates.


Forget gate -  Tries to remember only the important features and relationships overtime.
Input gate - Adds new information to old cell state at each time time step.
Output gate - Produces new output by taking into account old cell state and output at each time step.