How can I systematically learn about the theory of neural networks?
Good question, there is a lot of work in that field. The first part before saying which machine learning algorithm is better and why is defining the problem. Is the problem an optimisation, classification, anomaly detection problem because then you need to use the appropriate machine learning algorithms. Let's assume its an optimisation problem.

Is this problem, dynamic, or static. Is this time series data? So we need to understand the problem.

Each optimisation problem has a landscape or a fitness landscape. Computer science has some nice toy landscapes. 

There is a lot of work in determining the nature of the problem landscape. Have a look at K Malan's work.

Once you can identify the problem space and understand it's characteristics then you can start to identify what machine learning functions work well on what kind of landscapes. This is a totally different field of research. 

For example some researchers are working on how different evolutionary algorithms handle different landscapes, or neural networks handle different landscapes.

Start by exploring the types of machine learning problems.
Understand the complexity of the problem, followed by classification of machine learning algorithms for specific problem spaces.