What do neural networks learn first?
Potentially your looking for a description of the “vanishing or exploding gradient problem” in neural networks.

This post does a deep dive into this problem.  The author states the issue briefly as:


  an important observation: in at least some deep neural networks, the
  gradient tends to get smaller as we move backward through the hidden
  layers. This means that neurons in the earlier layers learn much more
  slowly than neurons in later layers.”


On a high level, we can look at the output of the neural network as a composition of functions $\hat{y} = f(g(h(x)))$. 

After obtaining the loss, $y - \hat{y}$, we use gradient descent to begin to updating the weights.  The weights in the deepest layer depend only on the derivative of the deepest function in the earlier composition, while shallow layers’ weights depend on the outer functions in the composition. 

The authors explain that this results in a derivative for the shallow layers that includes the multiplication of the deeper weights, which are usually &lt; 1 (in the vanishing gradient problem).  Multiplying many numbers &lt; 1 results in a very small number and a vanishing update to the weights in the shallow layers of the network.