Any explanation why multiple linear layers work better than a single linear layer in practice?
The key is that the layers of neurons in neural networks are not affine transformations. All commonly used neurons have some kind of non-linearity. The simplest of these is the Rectified Linear Unit (ReLU), which takes the form $y = x$ when $x &gt; 0$ and $y = 0$ for all other values, where $x$ is a weighted sum of the inputs to the neuron.