Is it possible to decode which neuron represent which feature and why does it represent it? In a neural network, Each neuron in the network represents some part of non-linear feature of the input. Ex: Like in mnist data, Consider the stem of number 9 is cut into multiple pieces and different part is represented by different neurons in the first hidden layer(Just an example from 3B1B neural networks video). My questions are:

What determines which neuron get to represent which part of the stem??

Is it possible that if we pass in the same input multiple times, each neuron can represent different part of the stem?? OR Is it that its all the magic of chain rule (i.e At the beginning, all neuron represent some trash feature and as updation of weights occur and then particular features have become synonymous to a particular neuron.) If so, How does this happen? 

Thanks in advance
Short answer: NO. 

Think about it, you could have 5 or 5000 neurons, and each neuron would represent a different thing in different models with different inputs. 

One of the problems of neural networks is that it is quite difficult to understand what is going inside them, they are a bit of a black box. Other techniques such as decision trees are much more easy to understand or explain to non-technical people.

The closest thing you could do is visualize the hidden layers and that might give you some sort of insight.