Problem with ADC in AVR XMEGA
Datasheet noise level (typical only) is given:

Reference voltage is not stated. If we assume it's 3.6V then the resolution is 0.8mV and we'd expect to see about 4 different values with the input shorted.
However with a non-zero DC voltage applied then other noise sources come into play. Depending on your conditions (and layout) 10 different values is not unexpected.
I think if you want decent quality audio capture you should use a separate audio ADC, maybe with I2S interface.
Alternatively, maybe you could capture 8 samples (say) and average them to reduce the noise by perhaps 2 or 3:1.