Is it possible to add "memory" to a neural network?
Yes it is possible. 
Recurrent neural networks address this issue. They are networks with loops in them, allowing information to persist.These loops make recurrent neural networks seem kind of mysterious. However, if you think a bit more, it turns out that they aren’t all that different than a normal neural network. A recurrent neural network can be thought of as multiple copies of the same network, each passing a message to a successor. 



Visit
http://colah.github.io/posts/2015-08-Understanding-LSTMs/ 
: This blog explains Recurrent Neural Network(RNN) and LSTM from scratch and also discuss few examples where LSTM is preferred over RNN.