Any explanation why multiple linear layers work better than a single linear layer in practice? It is a well-known math fact that composition of linear/affine transformations is still linear/affine. For a naive example,
$\textbf{A}_1\textbf{A}_2\textbf{x}$ is simply $\textbf{A}\textbf{x}$ where $\textbf{A}=\textbf{A}_1\textbf{A}_2$
Any one knows why in practice multiple linear layers tend to work better, even though it is mathematically equivalent to a single linear layer? Any reference is appreciated!
The key is that the layers of neurons in neural networks are not affine transformations. All commonly used neurons have some kind of non-linearity. The simplest of these is the Rectified Linear Unit (ReLU), which takes the form $y = x$ when $x &gt; 0$ and $y = 0$ for all other values, where $x$ is a weighted sum of the inputs to the neuron.