psd of an ac voltage signal
To compute the PSD, you'd need full amplitude/phase measurements of the noise signal. 

If by "255 ac measurements", you mean measurements of the RMS amplitude by a digital multimeter, then you do not have the necessary phase information. 

If by "255 ac measurements", you mean you are using an oscilloscope that is AC coupled, then you can consider the measurements to be a measurement of the voltage with the average value ("DC component") removed.   If we consider the voltage signal to be
$$
v_\text{total}(t) = v_\text{DC} + v_\text{AC}(t),
$$
$v_\text{DC}$ is going to be nearly entirely the "normal" contribution of the battery, and constant with time, and $v_\text{AC}(t)$ is going to vary with time and due entirely to thermal noise.  If your measurement is measuring $v_\text{AC}(t)$, then that is what you want anyways.

Once you have waveform samples, the PSD can be estimated using the periodogram (easiest, but noisy) or a more advanced method like Welch's or Blackman-Tukey method.