What are some resources with exercises related to neural networks?
The book Grokking Deep Learning, by Andrew Trask (a PhD student at Oxford University and a research scientist at DeepMind), a wonderful, clean, and plain-English discussion of the basic mechanics that go on under the hood of neural networks - from data flow to updating of weights.  It is written without a slant on normally-wonky math, the concepts are presented and then advanced at a digestible pace for anyone. 

Here are a few more possibly useful resources.


Neural Networks: Playground Exercises 
Getting Started With Deep Learning: Convolutional Neural Networks
Deep learning focuses on practical aspects of deep learning
First lab assignment in Deep learning
Deeplearning stanford