Autocorrelation of power spectrum
The autocorrelation function is the inverse Fourier Transform 
of the power spectrum. I have no idea where you got your formula
from or what your calculation is 
meant to be doing. Are you sure that you are reading the 
book/journal article/application note/website/Wikipedia or whatever
correctly?

In any case, if you want to compute an autocorrelation
as an ifft, you should be careful about making sure that
the power spectrum vector is set up right. If the 
autocorrelation function that you compute as the iffy
does not have a peak at the origin (or as the first
entry in the vector for MATLABi folks) and have the
symmetry property $R[i] = R[-i]$ in real life and
the appropriately translated version in MATLAB, something
is awry.