Understanding Layers in Recurrent Neural Networks for NLP
Its not like it just understands grammar.

In LSTMs the network tries to preserve the hidden states over time. By doing this they try to learn long-term dependencies in the language and relationships between words at variable distances.

LSTM does this by using its three famous gates.


Forget gate -  Tries to remember only the important features and relationships overtime.
Input gate - Adds new information to old cell state at each time time step.
Output gate - Produces new output by taking into account old cell state and output at each time step.