Deep learning basics
This link contains an amazing amount of deep learning literature.
Summarizing it here(going in the order a beginner ideally should)-
NOTE: All these resources mainly use python.

1) First of all, a basic knowledge of machine learning is required. I found Caltech's Learning from data to be ideal of all the machine learning courses available on the net.

Andrew Ng's Coursera course is pretty good too.

2) For Neural networks, nobody explains it better than Dr.Patrick Winston.
The assignments should be tried out for better understanding. They are in python.

3) For a better understanding of Neural Networks, Michael Nielsen's course should be done(as suggested by Alexey). It is pretty basic but it works.

4) For deep neural networks, and implementing them faster on GPUs, there are multiple frameworks available, such as Theano, Caffe, Pybrain, Torch,etc.
Out of these Theano provides a better low level functionality that allows its user to create custom NNs. It is a python library, so being able to use numpy,scikit-learn, matplotlib, scipy along with it is a big plus.
The deep learning tutorial written by Lisa Lab should be tried out for a better understanding of theano.

5) For Convolutional Neural Networks, follow andrej karpathy's tutorial.

6) For unsupervised learning, follow here and here.

7) For an intersection of deep learning and NLP, follow Richard Socher's class.

8) For LSTMs, read Hochreiter, S., &amp; Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780 and Graves, Alex. Supervised sequence labelling with recurrent neural networks. Vol. 385. Springer, 2012.

Here is LSTM's Theano code.