Which neuron represents which part of the input? In a neural network, each neuron represents some part of the input. For example, in the case of a MNIST digit, consider the stem of the number 9. Each neuron in the NN represents some part of this digit.


What determines which neuron will represent which part of the digit?
Is it possible that if we pass in the same input multiple times, each neuron can represent different parts of the digit?
How is this related to the back-propagation algorithm and chain rule? Is it the case that, before training the neural network, each neuron doesn't really represent anything of the input, and, as training proceeds, neurons start to represent some part of the input?
The determination of what each neuron represents is dictated by the initial weights of the network. As you may know, the common practice is to initialize the weights randomly. This means that given the same input data, the network may switch what each neuron means. It may also function differently, taking longer or shorter times to train. However, if you set the random seed to some constant, you can make the network deterministic.