Why is 4.096V so often used as a voltage reference?
It may or may not have historical significance, but if you have a perfect 12 bit DAC/ADC, each A/D count (or LSBit) corresponds to 1 mV, making some math easier.

FWIW, I personally see 2.5v being more common and I use internal bandgap references for non-critical measurements.