6883e621_ws___Machine_Learning__Theory___3
[0] Gradient Descent [1] Specify an architecture with free parameters and use gradient descent with respect to data to tune the parameters. [2] Relatively computationally tractable due to (a) modularity of gradient descent (b) directly optimizing the quantity you want to predict. [3] 
Finicky. There are issues with paremeter initialization, step size, and representation. It helps a great deal to have accumulated experience using this sort of system and there is little theoretical guidance. Overfitting is a significant issue.
All Models of Learning have Flaws « Machine Learning (Theory)
3