recurrent neural networks dependencies that do not matter in rnnsthe previous hidden state is the only previous memory the network remembers with lstmsin addition to the previous hidden statethe cell state is also remembered by the network the core concepts within lstm networks are the cell state and gates (inputoutputand forget gatesthese gates and the cell state include several operationssuch as sigmoid and tanh activationpointwise multiplication and additionand vector concatenation these operations help the cell state and gates to train the network to forget or propagate important information through the network cell states connect information throughout the network and thus help in passing long dependencies between sequences as and when needed an lstm can be described with the following set of equations note that the symbol denotes pointwise multiplication of two vectors--that isif [ and [ , , ]then [ , , the functions sg,and are non-linear activation functionsw and are weight matricesand the terms are bias terms wz rz yt - bz wi ri yt - pi - bi yt - ct - ot wo ro yt - po bo yt ot