Intuition behind the number of output neurons for a neural network
The reason 10 neurons work better than 4 is because it allows the network to encode all possible answers independently.

You could have 4 neurons and train the first one to encode the least significant bit, the second neuron the second-least significant bit, and so on. It would work.

Images of a 1 (bits: 0001) are quite like images of a 7 (bits 0111). Imagine an image where we know it’s a 1 or a 7, but we have no idea which one. With 4 outputs (as described) you would output (0, 0.5, 0.5, 1), meaning the correct answer is either 1, 3, 5, or 7. With 10 outputs, all classes would have 0 probability and 1 and 7 would have 50% each, which conveys more accurately what can be interpreted from the image.

You could have a single output that just outputs the value of the digit directly. But a 1 and 7 are now far apart in the output node (the whole range is from 0 to 9), so it’s impossible for this network to say “1 or 7”. It will probably say: 4 (midpoint between 1 and 7). That’s why it doesn’t work well.

Very poorly explained by “Nielsen’s book”, IMHO.