What order should I learn about Neural Networks?
I think, once you are covered with the common stuff, you can probably go on and study all kinds of neural network variants. 



The common stuff:

a) An undergraduate level Linear Algebra course -- covering matrix calculus. You might find this useful.

b) An undergraduate level study in statistical inference. Concepts from this topic will come up most of the time and you might have hard time getting around even though you understand the rest of the math. I would recommend this.

c) A starter book on neural networks. Ex- Neural networks by Raul Rojas.



After all these are covered you will certainly be ready for learning the variants of neural networks with ease. For LSTM I would recommend Alex Graves.