What is the lowest noise power source?
If you are sensitive to ultra low noise then the actual series resistance of the power source comes into play.  A simple resistance can generate noise which is dependent on temperature.

At first I would assume that a perfect capacitor would have less resistive noise then a battery that has some value of internal resistance.  Supercaps may have some drawbacks in relation to internal resistance too, (you would need to check the actual specifications). 

It turns out that even a capacitor can have some internal noise.  For some additional info see this wiki that includes calculations for both resistive noise and capacitive noise:   http://en.wikipedia.org/wiki/Johnson%E2%80%93Nyquist_noise

But if you are taking this battery current or supercap current through a regulator it is most certain that the regulator will generate more noise than the battery/supercap.